2022-10-20T23:25:56.197 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-20T23:25:56.198 INFO:teuthology.results:subset: '111/120000'
2022-10-20T23:25:56.199 INFO:teuthology.results:seed: '6855'
2022-10-21T22:11:29.741 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-21T22:11:29.769 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-10-22T10:11:35.890 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2022-10-22T10:11:54.868 INFO:scrape:Found 170 jobs
2022-10-22T10:13:21.594 INFO:scrape:Found 50 distinct failure reasons
2022-10-22T10:13:21.596 INFO:scrape:Failure: Test failure: test_fstop (tasks.cephfs.test_fstop.TestFSTop)
2022-10-22T10:13:21.597 INFO:scrape:1 jobs: ['7075785']
2022-10-22T10:13:21.597 INFO:scrape:suites: ['cluster/{1-node}', 'fs/top/{begin', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/ignorelist_health', 'supported-random-distros$/{centos_8}', 'tasks/fstop}']
2022-10-22T10:13:21.598 INFO:scrape:
2022-10-22T10:13:21.600 INFO:scrape:Dead: b'2022-10-21T16:28:15.992 DEBUG:teuthology.exit:Finished running handlers'
2022-10-22T10:13:21.600 INFO:scrape:16 jobs: ['7075730', '7075726', '7075720', '7075710', '7075923', '7075714', '7075722', '7075724', '7075732', '7075718', '7075736', '7075712', '7075738', '7075716', '7075729', '7075734']
2022-10-22T10:13:21.601 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}']
2022-10-22T10:13:21.602 INFO:scrape:suites union: ['1-client', '2-upgrade', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/pjd}}', '3-compat_client/pacific}}', 'centos_latest', 'clusters/1-mds-2-client', 'clusters/1-mds-2-client-micro', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/32bits/{begin', 'fs/functional/{begin', 'fs/mixed-clients/{begin', 'fs/multiclient/{begin', 'fs/permission/{begin', 'fs/thrash/multifs/{begin', 'fs/thrash/workloads/{begin', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/verify/{begin', 'fs/volumes/{begin', 'fs/workload/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'k-testing}', 'kclient-overrides/{distro/testing/{flavor/ubuntu_latest', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'multimds/no', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd-asserts}', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/no', 'scrub/yes', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/fsstress', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/mdtest}', 'tasks/quota}', 'tasks/recovery-fs}', 'tasks/scrub}', 'tasks/volumes/{overrides', 'tasks/{0-check-counter', 'tasks/{0-nautilus', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/snapshot}}', 'thrashosds-health}', 'validater/lockdep}', 'workunit/direct_io}', 'workunit/fs/misc}', 'wsync/{yes}}']
2022-10-22T10:13:21.603 INFO:scrape:
2022-10-22T10:13:21.605 INFO:scrape:Failure: Test failure: test_basic (tasks.cephfs.test_dump_tree.TestDumpTree)
2022-10-22T10:13:21.605 INFO:scrape:1 jobs: ['7075851']
2022-10-22T10:13:21.606 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/asok_dump_tree}']
2022-10-22T10:13:21.606 INFO:scrape:
2022-10-22T10:13:21.607 INFO:scrape:Failure: HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ffb576c0b38>: Failed to establish a new connection: [Errno 113] No route to host',))
2022-10-22T10:13:21.607 INFO:scrape:10 jobs: ['7075865', '7075881', '7075879', '7075859', '7075875', '7075887', '7075891', '7075885', '7075883', '7075889']
2022-10-22T10:13:21.608 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}']
2022-10-22T10:13:21.609 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-verify}', 'clusters/1-mds-2-client-micro', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'fs/32bits/{begin', 'fs/upgrade/volumes/import-legacy/{bluestore-bitmap', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/{0-nautilus', 'ubuntu_18.04}']
2022-10-22T10:13:21.609 INFO:scrape:
2022-10-22T10:13:21.610 INFO:scrape:Failure: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)
2022-10-22T10:13:21.611 INFO:scrape:1 jobs: ['7075781']
2022-10-22T10:13:21.612 INFO:scrape:suites: ['clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/shell/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs-shell}']
2022-10-22T10:13:21.613 INFO:scrape:
2022-10-22T10:13:21.615 INFO:scrape:Failure: Test failure: test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop)
2022-10-22T10:13:21.615 INFO:scrape:2 jobs: ['7075775', '7075947']
2022-10-22T10:13:21.616 INFO:scrape:suites intersection: ['conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}']
2022-10-22T10:13:21.617 INFO:scrape:suites union: ['clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}']
2022-10-22T10:13:21.618 INFO:scrape:
2022-10-22T10:13:21.619 INFO:scrape:Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage)
2022-10-22T10:13:21.619 INFO:scrape:1 jobs: ['7075913']
2022-10-22T10:13:21.619 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/damage}']
2022-10-22T10:13:21.620 INFO:scrape:
2022-10-22T10:13:21.620 INFO:scrape:Failure: "2022-10-21T20:41:24.541348+0000 mon.a (mon.0) 177 : cluster [WRN] Health check failed: Degraded data redundancy: 3/44 objects degraded (6.818%), 1 pg degraded (PG_DEGRADED)" in cluster log
2022-10-22T10:13:21.621 INFO:scrape:1 jobs: ['7075787']
2022-10-22T10:13:21.621 INFO:scrape:suites: ['clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/traceless/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'tasks/cfuse_workunit_suites_blogbench', 'traceless/50pc}']
2022-10-22T10:13:21.622 INFO:scrape:
2022-10-22T10:13:21.622 INFO:scrape:Failure: {'smithi203.front.sepia.ceph.com': {'_ansible_delegated_vars': {'ansible_connection': 'local', 'ansible_host': 'localhost', 'ansible_port': None, 'ansible_user': 'cm'}, '_ansible_no_log': False, 'attempts': 5, 'changed': False, 'cmd': ['/usr/bin/git', 'fetch', '--tags', '--force', 'origin'], 'invocation': {'module_args': {'accept_hostkey': False, 'archive': None, 'archive_prefix': None, 'bare': False, 'clone': True, 'depth': None, 'dest': '/home/teuthworker/.cache/src/keys', 'executable': None, 'force': True, 'gpg_whitelist': [], 'key_file': None, 'recursive': True, 'reference': None, 'refspec': None, 'remote': 'origin', 'repo': 'https://github.com/ceph/keys', 'separate_git_dir': None, 'ssh_opts': None, 'track_submodules': False, 'umask': None, 'update': True, 'verify_commit': False, 'version': 'main'}}, 'msg': "Failed to download remote objects and refs:  fatal: unable to access 'https://github.com/ceph/keys/': Could not resolve host: github.com\n"}, 'smithi088.front.sepia.ceph.com': {'_ansible_delegated_vars': {'ansible_connection': 'local', 'ansible_host': 'localhost', 'ansible_port': None, 'ansible_user': 'cm'}, '_ansible_no_log': False, 'attempts': 5, 'changed': False, 'cmd': ['/usr/bin/git', 'fetch', '--tags', '--force', 'origin'], 'invocation': {'module_args': {'accept_hostkey': False, 'archive': None, 'archive_prefix': None, 'bare': False, 'clone': True, 'depth': None, 'dest': '/home/teuthworker/.cache/src/keys', 'executable': None, 'force': True, 'gpg_whitelist': [], 'key_file': None, 'recursive': True, 'reference': None, 'refspec': None, 'remote': 'origin', 'repo': 'https://github.com/ceph/keys', 'separate_git_dir': None, 'ssh_opts': None, 'track_submodules': False, 'umask': None, 'update': True, 'verify_commit': False, 'version': 'main'}}, 'msg': "Failed to download remote objects and refs:  fatal: unable to access 'https://github.com/ceph/keys/': Could not resolve host: github.com\n"}}
2022-10-22T10:13:21.623 INFO:scrape:2 jobs: ['7075871', '7075869']
2022-10-22T10:13:21.624 INFO:scrape:suites intersection: ['conf/{client', 'distro/{ubuntu_latest}', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2022-10-22T10:13:21.625 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/permission/{begin', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/backtrace}', 'tasks/cfuse_workunit_misc}']
2022-10-22T10:13:21.625 INFO:scrape:
2022-10-22T10:13:21.625 INFO:scrape:Failure: Test failure: test_abort_conn (tasks.cephfs.test_client_recovery.TestClientRecovery)
2022-10-22T10:13:21.626 INFO:scrape:1 jobs: ['7075897']
2022-10-22T10:13:21.626 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/client-recovery}']
2022-10-22T10:13:21.627 INFO:scrape:
2022-10-22T10:13:21.627 INFO:scrape:Failure: Command failed on smithi111 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8977ca66-5183-11ed-8438-001a4aab830c -e sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\''
2022-10-22T10:13:21.628 INFO:scrape:6 jobs: ['7075807', '7075673', '7075656', '7075919', '7075616', '7075745']
2022-10-22T10:13:21.628 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2022-10-22T10:13:21.629 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2022-10-22T10:13:21.629 INFO:scrape:
2022-10-22T10:13:21.630 INFO:scrape:Failure: Test failure: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity)
2022-10-22T10:13:21.630 INFO:scrape:2 jobs: ['7075845', '7075640']
2022-10-22T10:13:21.631 INFO:scrape:suites intersection: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{ignorelist_health', 'tasks/failover}']
2022-10-22T10:13:21.631 INFO:scrape:suites union: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mon-debug}', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/failover}']
2022-10-22T10:13:21.632 INFO:scrape:
2022-10-22T10:13:21.632 INFO:scrape:Failure: Command failed on smithi188 with status 1: 'sudo yum install -y kernel'
2022-10-22T10:13:21.632 INFO:scrape:2 jobs: ['7075877', '7075763']
2022-10-22T10:13:21.633 INFO:scrape:suites intersection: []
2022-10-22T10:13:21.634 INFO:scrape:suites union: ['1-mount', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-workload/dbench', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'distro/centos_latest}', 'fs/cephadm/multivolume/{0-start', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4']
2022-10-22T10:13:21.634 INFO:scrape:
2022-10-22T10:13:21.635 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)
2022-10-22T10:13:21.635 INFO:scrape:1 jobs: ['7075821']
2022-10-22T10:13:21.635 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/acls}']
2022-10-22T10:13:21.636 INFO:scrape:
2022-10-22T10:13:21.636 INFO:scrape:Failure: reached maximum tries (60) after waiting for 600 seconds
2022-10-22T10:13:21.637 INFO:scrape:2 jobs: ['7075706', '7075708']
2022-10-22T10:13:21.637 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2022-10-22T10:13:21.638 INFO:scrape:suites union: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg-warn}', 'rhel_8}', 'roles', 'tasks/pool-perm}', 'tasks/{0-from/v16.2.4']
2022-10-22T10:13:21.638 INFO:scrape:
2022-10-22T10:13:21.639 INFO:scrape:Failure: Command failed on smithi154 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev'
2022-10-22T10:13:21.639 INFO:scrape:1 jobs: ['7075853']
2022-10-22T10:13:21.640 INFO:scrape:suites: ['2-workunit/cfuse_workunit_trivial_sync}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2022-10-22T10:13:21.640 INFO:scrape:
2022-10-22T10:13:21.641 INFO:scrape:Crash: Command failed on smithi063 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'
2022-10-22T10:13:21.641 INFO:scrape:ceph version 16.2.10-1119-gd88f759b (d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a) pacific (stable)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7f60b1298b20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f60b355c779]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x279942) [0x7f60b355c942]
 6: (Monitor::~Monitor()+0xe40) [0x558d85c779c0]
 7: (Monitor::~Monitor()+0xd) [0x558d85c77a1d]
 8: main()
 9: __libc_start_main()
 10: _start()
2022-10-22T10:13:21.641 INFO:scrape:1 jobs: ['7075801']
2022-10-22T10:13:21.642 INFO:scrape:suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/blogbench}', 'wsync/{yes}}']
2022-10-22T10:13:21.642 INFO:scrape:
2022-10-22T10:13:21.643 INFO:scrape:Failure: "2022-10-21T00:51:24.054109+0000 mds.f (mds.0) 29 : cluster [WRN] client.4870 isn't responding to mclientcaps(revoke), ino 0x1000000f36c pending pAsxLsXsxFscr issued pAsxLsXsxFsxcrwb, sent 304.986632 seconds ago" in cluster log
2022-10-22T10:13:21.643 INFO:scrape:1 jobs: ['7075620']
2022-10-22T10:13:21.644 INFO:scrape:suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-10-22T10:13:21.644 INFO:scrape:
2022-10-22T10:13:21.644 INFO:scrape:Failure: Test failure: test_barrier (tasks.cephfs.test_full.TestClusterFull)
2022-10-22T10:13:21.645 INFO:scrape:1 jobs: ['7075667']
2022-10-22T10:13:21.645 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/mds-full}']
2022-10-22T10:13:21.646 INFO:scrape:
2022-10-22T10:13:21.646 INFO:scrape:Failure: Test failure: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules)
2022-10-22T10:13:21.647 INFO:scrape:1 jobs: ['7075747']
2022-10-22T10:13:21.647 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/snap-schedule}']
2022-10-22T10:13:21.648 INFO:scrape:
2022-10-22T10:13:21.648 INFO:scrape:Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan)
2022-10-22T10:13:21.649 INFO:scrape:2 jobs: ['7075618', '7075917']
2022-10-22T10:13:21.649 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/data-scan}']
2022-10-22T10:13:21.649 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/data-scan}']
2022-10-22T10:13:21.650 INFO:scrape:
2022-10-22T10:13:21.650 INFO:scrape:Failure: Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool)
2022-10-22T10:13:21.651 INFO:scrape:1 jobs: ['7075841']
2022-10-22T10:13:21.651 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/alternate-pool}']
2022-10-22T10:13:21.651 INFO:scrape:
2022-10-22T10:13:21.652 INFO:scrape:Failure: Test failure: test_inject_to_empty (tasks.cephfs.test_journal_repair.TestJournalRepair)
2022-10-22T10:13:21.652 INFO:scrape:2 jobs: ['7075953', '7075650']
2022-10-22T10:13:21.653 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/journal-repair}']
2022-10-22T10:13:21.653 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/journal-repair}']
2022-10-22T10:13:21.654 INFO:scrape:
2022-10-22T10:13:21.654 INFO:scrape:Failure: Test failure: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
2022-10-22T10:13:21.655 INFO:scrape:1 jobs: ['7075622']
2022-10-22T10:13:21.655 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/clone}}']
2022-10-22T10:13:21.655 INFO:scrape:
2022-10-22T10:13:21.656 INFO:scrape:Failure: Test failure: test_15303 (tasks.cephfs.test_volume_client.TestVolumeClient)
2022-10-22T10:13:21.656 INFO:scrape:1 jobs: ['7075793']
2022-10-22T10:13:21.657 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volume-client}']
2022-10-22T10:13:21.657 INFO:scrape:
2022-10-22T10:13:21.658 INFO:scrape:Failure: {'smithi085.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 24, 'changed': False, 'invocation': {'module_args': {'comment': None, 'exclusive': False, 'follow': False, 'key': 'https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub', 'key_options': None, 'manage_dir': True, 'path': None, 'state': 'present', 'user': 'ubuntu', 'validate_certs': True}}, 'msg': 'Error getting key from: https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub'}, 'smithi017.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts': 24, 'changed': False, 'invocation': {'module_args': {'comment': None, 'exclusive': False, 'follow': False, 'key': 'https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub', 'key_options': None, 'manage_dir': True, 'path': None, 'state': 'present', 'user': 'ubuntu', 'validate_certs': True}}, 'msg': 'Error getting key from: https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub'}}
2022-10-22T10:13:21.658 INFO:scrape:1 jobs: ['7075873']
2022-10-22T10:13:21.659 INFO:scrape:suites: ['2-workunit/ffsb}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2022-10-22T10:13:21.659 INFO:scrape:
2022-10-22T10:13:21.660 INFO:scrape:Failure: Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)
2022-10-22T10:13:21.660 INFO:scrape:3 jobs: ['7075700', '7075905', '7075777']
2022-10-22T10:13:21.660 INFO:scrape:suites intersection: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'osd}', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2022-10-22T10:13:21.661 INFO:scrape:suites union: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mon-debug}', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2022-10-22T10:13:21.661 INFO:scrape:
2022-10-22T10:13:21.662 INFO:scrape:Failure: Test failure: test_ephemeral_pin_dist_conf_off (tasks.cephfs.test_exports.TestEphemeralPins)
2022-10-22T10:13:21.671 INFO:scrape:2 jobs: ['7075925', '7075624']
2022-10-22T10:13:21.672 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/exports}']
2022-10-22T10:13:21.673 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/exports}']
2022-10-22T10:13:21.673 INFO:scrape:
2022-10-22T10:13:21.674 INFO:scrape:Failure: Test failure: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub)
2022-10-22T10:13:21.676 INFO:scrape:2 jobs: ['7075933', '7075630']
2022-10-22T10:13:21.677 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/forward-scrub}']
2022-10-22T10:13:21.678 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/forward-scrub}']
2022-10-22T10:13:21.679 INFO:scrape:
2022-10-22T10:13:21.680 INFO:scrape:Failure: {'smithi045.front.sepia.ceph.com': {'_ansible_no_log': False, 'cache_update_time': 1666387794, 'cache_updated': False, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': True, 'force_apt_get': False, 'install_recommends': None, 'name': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'python-dev', 'libfcgi0ldbl', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'only_upgrade': False, 'package': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'python-dev', 'libfcgi0ldbl', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache': None, 'update_cache_retries': 5, 'update_cache_retry_max_delay': 12, 'upgrade': None}}, 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"   --force-yes   install \'qemu-system-x86\' \'git-core\'\' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git_2.25.1-1ubuntu3.6_amd64.deb  Could not connect to archive.ubuntu.com:80 (91.189.91.38). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (91.189.91.39). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::19), connection timed out Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::16), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::18), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::15), connection timed out\nE: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librados2_15.2.16-0ubuntu0.20.04.1_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librbd1_15.2.16-0ubuntu0.20.04.1_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-block-extra_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-system-common_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-system-x86_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-utils_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n', 'rc': 100, 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git_2.25.1-1ubuntu3.6_amd64.deb  Could not connect to archive.ubuntu.com:80 (91.189.91.38). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (91.189.91.39). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::19), connection timed out Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::16), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::18), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::15), connection timed out\nE: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librados2_15.2.16-0ubuntu0.20.04.1_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librbd1_15.2.16-0ubuntu0.20.04.1_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-block-extra_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-system-common_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-system-x86_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-utils_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n', 'stderr_lines': ['W: --force-yes is deprecated, use one of the options starting with --allow instead.', 'E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git_2.25.1-1ubuntu3.6_amd64.deb  Could not connect to archive.ubuntu.com:80 (91.189.91.38). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (91.189.91.39). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::19), connection timed out Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::16), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::18), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::15), connection timed out', 'E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librados2_15.2.16-0ubuntu0.20.04.1_amd64.deb  Unable to connect to archive.ubuntu.com:http:', 'E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librbd1_15.2.16-0ubuntu0.20.04.1_amd64.deb  Unable to connect to archive.ubuntu.com:http:', 'E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-block-extra_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:', 'E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-system-common_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:', 'E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-system-x86_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:', 'E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-utils_4.2-3ubuntu6.23_amd64.deb  Unable to connect to archive.ubuntu.com:http:', 'E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?'], 'stdout': 'Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n  librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n  git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n  gitweb git-cvs git-mediawiki git-svn samba vde2 debootstrap\nThe following NEW packages will be installed:\n  librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n  qemu-utils\nThe following packages will be upgraded:\n  git\n1 upgraded, 6 newly installed, 0 to remove and 13 not upgraded.\nNeed to get 18.2 MB of archives.\nAfter this operation, 62.7 MB of additional disk space will be used.\nIgn:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.6\nErr:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.16-0ubuntu0.20.04.1\n  Unable to connect to archive.ubuntu.com:http:\nErr:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.16-0ubuntu0.20.04.1\n  Unable to connect to archive.ubuntu.com:http:\nIgn:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.23\nIgn:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.23\nIgn:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.23\nIgn:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.23\nErr:1 http://security.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.6\n  Could not connect to archive.ubuntu.com:80 (91.189.91.38). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (91.189.91.39). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::19), connection timed out Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::16), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::18), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::15), connection timed out\nErr:4 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.23\n  Unable to connect to archive.ubuntu.com:http:\nErr:5 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.23\n  Unable to connect to archive.ubuntu.com:http:\nErr:6 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.23\n  Unable to connect to archive.ubuntu.com:http:\nErr:7 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.23\n  Unable to connect to archive.ubuntu.com:http:\n', 'stdout_lines': ['Reading package lists...', 'Building dependency tree...', 'Reading state information...', 'The following additional packages will be installed:', '  librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils', 'Suggested packages:', '  git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk', '  gitweb git-cvs git-mediawiki git-svn samba vde2 debootstrap', 'The following NEW packages will be installed:', '  librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86', '  qemu-utils', 'The following packages will be upgraded:', '  git', '1 upgraded, 6 newly installed, 0 to remove and 13 not upgraded.', 'Need to get 18.2 MB of archives.', 'After this operation, 62.7 MB of additional disk space will be used.', 'Ign:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.6', 'Err:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.16-0ubuntu0.20.04.1', '  Unable to connect to archive.ubuntu.com:http:', 'Err:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.16-0ubuntu0.20.04.1', '  Unable to connect to archive.ubuntu.com:http:', 'Ign:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.23', 'Ign:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.23', 'Ign:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.23', 'Ign:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.23', 'Err:1 http://security.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.6', '  Could not connect to archive.ubuntu.com:80 (91.189.91.38). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (91.189.91.39). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::19), connection timed out Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::16), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::18), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::15), connection timed out', 'Err:4 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.23', '  Unable to connect to archive.ubuntu.com:http:', 'Err:5 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.23', '  Unable to connect to archive.ubuntu.com:http:', 'Err:6 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.23', '  Unable to connect to archive.ubuntu.com:http:', 'Err:7 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.23', '  Unable to connect to archive.ubuntu.com:http:']}, 'smithi137.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'cmd': ['/usr/bin/pip3', 'install', '-U', 'git+https://github.com/sebastian-philipp/registries-conf-ctl'], 'invocation': {'module_args': {'chdir': None, 'editable': False, 'executable': 'pip3', 'extra_args': None, 'name': ['git+https://github.com/sebastian-philipp/registries-conf-ctl'], 'requirements': None, 'state': 'latest', 'umask': None, 'version': None, 'virtualenv': None, 'virtualenv_command': 'virtualenv', 'virtualenv_python': None, 'virtualenv_site_packages': False}}, 'msg': "stdout: Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n  Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-jb9nktur\n\n:stderr:   Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-jb9nktur\n  fatal: unable to access 'https://github.com/sebastian-philipp/registries-conf-ctl/': Could not resolve host: github.com\nERROR: Command errored out with exit status 128: git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-jb9nktur Check the logs for full command output.\n"}, 'smithi057.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'cmd': ['/usr/bin/pip3', 'install', '-U', 'git+https://github.com/sebastian-philipp/registries-conf-ctl'], 'invocation': {'module_args': {'chdir': None, 'editable': False, 'executable': 'pip3', 'extra_args': None, 'name': ['git+https://github.com/sebastian-philipp/registries-conf-ctl'], 'requirements': None, 'state': 'latest', 'umask': None, 'version': None, 'virtualenv': None, 'virtualenv_command': 'virtualenv', 'virtualenv_python': None, 'virtualenv_site_packages': False}}, 'msg': "stdout: Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n  Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-d156ofl8\n\n:stderr:   Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-d156ofl8\n  fatal: unable to access 'https://github.com/sebastian-philipp/registries-conf-ctl/': Could not resolve host: github.com\nERROR: Command errored out with exit status 128: git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-d156ofl8 Check the logs for full command output.\n"}}
2022-10-22T10:13:21.681 INFO:scrape:1 jobs: ['7075863']
2022-10-22T10:13:21.682 INFO:scrape:suites: ['clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'ranks/3', 'scrub/no', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/fs/norstats}', 'wsync/{no}}']
2022-10-22T10:13:21.683 INFO:scrape:
2022-10-22T10:13:21.683 INFO:scrape:Failure: Test failure: test_dump_subtrees (tasks.cephfs.test_admin.TestAdminCommandDumpTree)
2022-10-22T10:13:21.684 INFO:scrape:1 jobs: ['7075829']
2022-10-22T10:13:21.685 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/admin}']
2022-10-22T10:13:21.686 INFO:scrape:
2022-10-22T10:13:21.687 INFO:scrape:Failure: Command failed on smithi006 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a'
2022-10-22T10:13:21.688 INFO:scrape:1 jobs: ['7075861']
2022-10-22T10:13:21.689 INFO:scrape:suites: ['2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2022-10-22T10:13:21.689 INFO:scrape:
2022-10-22T10:13:21.690 INFO:scrape:Failure: Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration)
2022-10-22T10:13:21.691 INFO:scrape:1 jobs: ['7075797']
2022-10-22T10:13:21.692 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/test_journal_migration}']
2022-10-22T10:13:21.693 INFO:scrape:
2022-10-22T10:13:21.694 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)
2022-10-22T10:13:21.694 INFO:scrape:1 jobs: ['7075658']
2022-10-22T10:13:21.695 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/mds-flush}']
2022-10-22T10:13:21.695 INFO:scrape:
2022-10-22T10:13:21.696 INFO:scrape:Failure: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring)
2022-10-22T10:13:21.697 INFO:scrape:2 jobs: ['7075769', '7075791']
2022-10-22T10:13:21.698 INFO:scrape:suites intersection: ['mount/fuse']
2022-10-22T10:13:21.699 INFO:scrape:suites union: ['centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/whitelist_health', 'overrides/{whitelist_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/mirror}', 'tasks/mirror}}']
2022-10-22T10:13:21.700 INFO:scrape:
2022-10-22T10:13:21.701 INFO:scrape:Failure: Test failure: test_backtrace_repair (tasks.cephfs.test_auto_repair.TestMDSAutoRepair)
2022-10-22T10:13:21.702 INFO:scrape:1 jobs: ['7075857']
2022-10-22T10:13:21.703 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/auto-repair}']
2022-10-22T10:13:21.704 INFO:scrape:
2022-10-22T10:13:21.704 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2022-10-22T10:13:21.705 INFO:scrape:1 jobs: ['7075664']
2022-10-22T10:13:21.705 INFO:scrape:suites: ['2-workunit/suites/fsstress}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2022-10-22T10:13:21.706 INFO:scrape:
2022-10-22T10:13:21.706 INFO:scrape:Failure: Test failure: test_binary_metafile_on_legacy_to_v1_upgrade (tasks.cephfs.test_volumes.TestMisc)
2022-10-22T10:13:21.707 INFO:scrape:2 jobs: ['7075668', '7075929']
2022-10-22T10:13:21.707 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/misc}}']
2022-10-22T10:13:21.708 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/misc}}']
2022-10-22T10:13:21.708 INFO:scrape:
2022-10-22T10:13:21.709 INFO:scrape:Failure: Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation)
2022-10-22T10:13:21.709 INFO:scrape:2 jobs: ['7075937', '7075634']
2022-10-22T10:13:21.710 INFO:scrape:suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/fragment}']
2022-10-22T10:13:21.710 INFO:scrape:suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/fragment}']
2022-10-22T10:13:21.711 INFO:scrape:
2022-10-22T10:13:21.711 INFO:scrape:Failure: Test failure: test_dir_deletion (tasks.cephfs.test_strays.TestStrays)
2022-10-22T10:13:21.711 INFO:scrape:1 jobs: ['7075757']
2022-10-22T10:13:21.712 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/strays}']
2022-10-22T10:13:21.712 INFO:scrape:
2022-10-22T10:13:21.713 INFO:scrape:Failure: Test failure: test_apply_tag (tasks.cephfs.test_multimds_misc.TestScrub2)
2022-10-22T10:13:21.714 INFO:scrape:1 jobs: ['7075688']
2022-10-22T10:13:21.716 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/multimds_misc}']
2022-10-22T10:13:21.717 INFO:scrape:
2022-10-22T10:13:21.717 INFO:scrape:Failure: Test failure: test_mount_conn_close (tasks.cephfs.test_sessionmap.TestSessionMap)
2022-10-22T10:13:21.718 INFO:scrape:1 jobs: ['7075741']
2022-10-22T10:13:21.718 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/sessionmap}']
2022-10-22T10:13:21.718 INFO:scrape:
2022-10-22T10:13:21.719 INFO:scrape:Failure: Test failure: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable)
2022-10-22T10:13:21.719 INFO:scrape:1 jobs: ['7075696']
2022-10-22T10:13:21.720 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/openfiletable}']
2022-10-22T10:13:21.721 INFO:scrape:
2022-10-22T10:13:21.721 INFO:scrape:Failure: {'smithi110.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': 'ceph/python3-asyncssh', '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': ['dnf', '-y', 'copr', 'enable', 'ceph/python3-asyncssh'], 'delta': '0:00:15.346051', 'end': '2022-10-21 21:31:10.601574', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'dnf -y copr enable ceph/python3-asyncssh', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': 'ceph/python3-asyncssh', 'msg': 'non-zero return code', 'rc': 1, 'start': '2022-10-21 21:30:55.255523', 'stderr': 'Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.\nError: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Name or service not known', 'stderr_lines': ['Enabling a Copr repository. Please note that this repository is not part', 'of the main distribution, and quality may vary.', '', 'The Fedora Project does not exercise any power over the contents of', 'this repository beyond the rules outlined in the Copr FAQ at', '<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,', 'and packages are not held to any quality or security level.', '', 'Please do not file bug reports about these packages in Fedora', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Name or service not known'], 'stdout': '', 'stdout_lines': []}], 'warnings': ["Consider using the dnf module rather than running 'dnf'.  If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]}, 'smithi175.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'cmd': ['/bin/pip3', 'install', '-U', 'git+https://github.com/sebastian-philipp/registries-conf-ctl'], 'invocation': {'module_args': {'chdir': None, 'editable': False, 'executable': 'pip3', 'extra_args': None, 'name': ['git+https://github.com/sebastian-philipp/registries-conf-ctl'], 'requirements': None, 'state': 'latest', 'umask': None, 'version': None, 'virtualenv': None, 'virtualenv_command': 'virtualenv', 'virtualenv_python': None, 'virtualenv_site_packages': False}}, 'msg': 'stdout: Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n  Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-_8z5oo5m-build\nfatal: unable to access \'https://github.com/sebastian-philipp/registries-conf-ctl/\': Could not resolve host: github.com\n\n:stderr: WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\nCommand "git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-_8z5oo5m-build" failed with error code 128 in None\n'}}
2022-10-22T10:13:21.722 INFO:scrape:1 jobs: ['7075855']
2022-10-22T10:13:21.722 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/verify/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/lockdep}']
2022-10-22T10:13:21.723 INFO:scrape:
2022-10-22T10:13:21.724 INFO:scrape:Failure: Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)
2022-10-22T10:13:21.724 INFO:scrape:1 jobs: ['7075843']
2022-10-22T10:13:21.725 INFO:scrape:suites: ['1-client-upgrade', '2-client-sanity}}', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus']
2022-10-22T10:13:21.726 INFO:scrape:
2022-10-22T10:13:21.726 INFO:scrape:Failure: Command failed on smithi044 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel'
2022-10-22T10:13:21.727 INFO:scrape:1 jobs: ['7075867']
2022-10-22T10:13:21.727 INFO:scrape:suites: ['clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/ino_release_cb}']
2022-10-22T10:13:21.728 INFO:scrape:
2022-10-22T10:13:21.728 INFO:scrape:Failure: Test failure: test_volume_create (tasks.cephfs.test_volumes.TestVolumes)
2022-10-22T10:13:21.728 INFO:scrape:1 jobs: ['7075831']
2022-10-22T10:13:21.729 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}']
2022-10-22T10:13:21.730 INFO:scrape:
2022-10-22T10:13:21.731 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi085 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2022-10-22T10:13:21.732 INFO:scrape:1 jobs: ['7075702']
2022-10-22T10:13:21.733 INFO:scrape:suites: ['clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/snaps/{begin', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}']
2022-10-22T10:13:21.734 INFO:scrape:
2022-10-22T10:13:21.735 INFO:scrape:Failure: Test failure: test_client_metrics_and_metadata (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
2022-10-22T10:13:21.736 INFO:scrape:1 jobs: ['7075681']
2022-10-22T10:13:21.737 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'tasks/metrics}']
2022-10-22T10:13:21.738 INFO:scrape:
2022-10-22T10:13:21.739 INFO:scrape:Failure: Test failure: test_allow_new_snaps_config (tasks.cephfs.test_snapshots.TestSnapshots)
2022-10-22T10:13:21.739 INFO:scrape:1 jobs: ['7075751']
2022-10-22T10:13:21.740 INFO:scrape:suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/snapshots}']
2022-10-22T10:13:21.740 INFO:scrape:
2022-10-22T10:13:21.754 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/
failed:  72
dead:    20
running: 1
waiting: 0
queued:  0
passed:  77


Fail
=================================================================
[7075616]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:23:13
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075616/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075616/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9b6ec63d783c41d5bcd3cb597cb1929f

    Command failed on smithi032 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    b0ab0cbe-50d0-11ed-8437-001a4aab830c -e
    sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\''

[7075618]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
-----------------------------------------------------------------
time:   00:15:56
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075618/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075618/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b3000e53684e4d5494bc26bcb9fba3ae

    Test failure: test_fragmented_injection
    (tasks.cephfs.test_data_scan.TestDataScan)

[7075620]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   02:36:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075620/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075620/

    "2022-10-21T00:51:24.054109+0000 mds.f (mds.0) 29 : cluster [WRN]
    client.4870 isn't responding to mclientcaps(revoke), ino 0x1000000f36c
    pending pAsxLsXsxFscr issued pAsxLsXsxFsxcrwb, sent 304.986632 seconds ago"
    in cluster log

[7075622]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   00:11:06
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075622/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075622/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1e3c81c7aae14b638b6172b1f5d8772e

    Test failure: test_clone_failure_status_failed
    (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)

[7075624]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
-----------------------------------------------------------------
time:   00:13:48
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075624/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075624/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4c8f85e3ceed4668b0875fb565b69d75

    Test failure: test_ephemeral_pin_dist_conf_off
    (tasks.cephfs.test_exports.TestEphemeralPins)

[7075630]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:10:41
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075630/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075630/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4296fba619b643efa059bb2a3a14cf2b

    Test failure: test_apply_tag
    (tasks.cephfs.test_forward_scrub.TestForwardScrub)

[7075634]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
-----------------------------------------------------------------
time:   00:11:09
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075634/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075634/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8518f4589426485fb50307974d07564c

    Test failure: test_deep_split
    (tasks.cephfs.test_fragment.TestFragmentation)

[7075640]  fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:14:16
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075640/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075640/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b8c0f0d066b642069e9a3d466712a5de

    Test failure: test_join_fs_drop
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7075650]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
-----------------------------------------------------------------
time:   00:16:44
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075650/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075650/
sentry: https://sentry.ceph.com/organizations/ceph/?query=54249f76606448c3ac2823b0fdef0dc5

    Test failure: test_inject_to_empty
    (tasks.cephfs.test_journal_repair.TestJournalRepair)

[7075656]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:23:52
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075656/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075656/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6c0785d7c8094705b38a97f45c7dc12e

    Command failed on smithi046 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    5d68bd9c-50e1-11ed-8437-001a4aab830c -e
    sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\''

[7075658]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush}
-----------------------------------------------------------------
time:   00:15:23
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075658/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075658/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a6f5a336b29a473889894c1c9db0ad55

    Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)

[7075664]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   06:34:24
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075664/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075664/
sentry: https://sentry.ceph.com/organizations/ceph/?query=645f0ddf03ae4f70972ecc7adb0a78eb

    Command failed (workunit test suites/fsstress.sh) on smithi063 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7075667]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
-----------------------------------------------------------------
time:   00:10:34
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075667/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075667/
sentry: https://sentry.ceph.com/organizations/ceph/?query=86b80c43ae7d402eb73924a51995e849

    Test failure: test_barrier (tasks.cephfs.test_full.TestClusterFull)

[7075668]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
-----------------------------------------------------------------
time:   00:16:59
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075668/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075668/
sentry: https://sentry.ceph.com/organizations/ceph/?query=51a192b8994d49f7bfd9fc5176f9d25f

    Test failure: test_binary_metafile_on_legacy_to_v1_upgrade
    (tasks.cephfs.test_volumes.TestMisc)

[7075673]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:24:22
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075673/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075673/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c90e32c827ee42138c3e2d359808f1ff

    Command failed on smithi046 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    13b29cde-50e8-11ed-8437-001a4aab830c -e
    sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\''

[7075681]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics}
-----------------------------------------------------------------
time:   00:16:13
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075681/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075681/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0895a471afdf4ec18ef410ea75a8c789

    Test failure: test_client_metrics_and_metadata
    (tasks.cephfs.test_mds_metrics.TestMDSMetrics)

[7075688]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
-----------------------------------------------------------------
time:   00:14:28
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075688/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075688/
sentry: https://sentry.ceph.com/organizations/ceph/?query=12d43d2250b845cbbf64a4f63226c009

    Test failure: test_apply_tag (tasks.cephfs.test_multimds_misc.TestScrub2)

[7075696]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
-----------------------------------------------------------------
time:   00:09:07
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075696/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075696/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ff5d6990c9f04811935b4cd45456f9ba

    Test failure: test_max_items_per_obj
    (tasks.cephfs.test_openfiletable.OpenFileTable)

[7075700]  fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:10:27
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075700/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075700/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fc8f4c0b5ab047ddb3f4bfda26b78353

    Test failure: test_mount_all_caps_absent
    (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)

[7075702]  fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:16:17
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075702/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075702/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ce73e686ce4b4e2e8d3e24c83b03c993

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi085
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7075706]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
-----------------------------------------------------------------
time:   04:37:57
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075706/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075706/
sentry: https://sentry.ceph.com/organizations/ceph/?query=990cf0b1e62442209e297c2b75f00e86

    reached maximum tries (60) after waiting for 600 seconds

[7075708]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   06:51:38
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075708/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075708/
sentry: https://sentry.ceph.com/organizations/ceph/?query=818431ceafd64f988c59dcaca5911ee3

    reached maximum tries (60) after waiting for 600 seconds

[7075741]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap}
-----------------------------------------------------------------
time:   11:58:28
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075741/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075741/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d4622587139e4856a8a834fc58bb9918

    Test failure: test_mount_conn_close
    (tasks.cephfs.test_sessionmap.TestSessionMap)

[7075745]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   05:58:21
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075745/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075745/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8d6d2625cd6e43a49a85f81e702d9ce6

    Command failed on smithi093 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    620edfc6-5180-11ed-8438-001a4aab830c -e
    sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\''

[7075747]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   05:12:58
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075747/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075747/
sentry: https://sentry.ceph.com/organizations/ceph/?query=98db39609afb4cfa81526b3f067d66d5

    Test failure: test_concurrent_snap_creates
    (tasks.cephfs.test_snap_schedules.TestSnapSchedules)

[7075751]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
-----------------------------------------------------------------
time:   05:11:22
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075751/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075751/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9c365c4b6e854b47baf961ddf37a1fdf

    Test failure: test_allow_new_snaps_config
    (tasks.cephfs.test_snapshots.TestSnapshots)

[7075757]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
-----------------------------------------------------------------
time:   05:11:48
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075757/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075757/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6384416458964918968c71bed962fef7

    Test failure: test_dir_deletion (tasks.cephfs.test_strays.TestStrays)

[7075763]  fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest}
-----------------------------------------------------------------
time:   05:13:02
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075763/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075763/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c053e518441f4f6a9acff0ac79917fd1

    Command failed on smithi171 with status 5: 'sudo systemctl stop
    ceph-912bd64c-5180-11ed-8438-001a4aab830c@mon.smithi171'

[7075769]  fs/mirror/{begin cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror}
-----------------------------------------------------------------
time:   04:58:05
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075769/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075769/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b8be42c5f5da4d8081678e1169b6ef79

    Test failure: test_add_ancestor_and_child_directory
    (tasks.cephfs.test_mirroring.TestMirroring)

[7075775]  fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   04:37:26
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075775/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075775/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f56ed5d5623c4934adf842f3cc6eb28f

    Test failure: test_drop_cache_command
    (tasks.cephfs.test_misc.TestCacheDrop)

[7075777]  fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   04:07:24
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075777/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075777/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0e5c2413329948a48c54d1cfeb8011bd

    Test failure: test_mount_all_caps_absent
    (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)

[7075781]  fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell}
-----------------------------------------------------------------
time:   03:46:46
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075781/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075781/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a358d491cab84c56b5a4a4546b3eb8b6

    Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)

[7075785]  fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop}
-----------------------------------------------------------------
time:   03:23:32
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075785/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075785/
sentry: https://sentry.ceph.com/organizations/ceph/?query=422420b8e055474bb8b3ea36f0c02b00

    Test failure: test_fstop (tasks.cephfs.test_fstop.TestFSTop)

[7075787]  fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
-----------------------------------------------------------------
time:   03:36:04
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075787/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075787/

    "2022-10-21T20:41:24.541348+0000 mon.a (mon.0) 177 : cluster [WRN] Health
    check failed: Degraded data redundancy: 3/44 objects degraded (6.818%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[7075791]  fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   02:44:43
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075791/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075791/
sentry: https://sentry.ceph.com/organizations/ceph/?query=11338042dfcb4e43864b60c34e492d92

    Test failure: test_add_ancestor_and_child_directory
    (tasks.cephfs.test_mirroring.TestMirroring)

[7075793]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volume-client}
-----------------------------------------------------------------
time:   00:09:08
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075793/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075793/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3831be2eecd444d2890427de4e124923

    Test failure: test_15303 (tasks.cephfs.test_volume_client.TestVolumeClient)

[7075797]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
-----------------------------------------------------------------
time:   00:11:34
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075797/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075797/
sentry: https://sentry.ceph.com/organizations/ceph/?query=adb898c6a1d44014bfa304932c94e024

    Test failure: test_journal_migration
    (tasks.cephfs.test_journal_migration.TestJournalMigration)

[7075801]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}}
-----------------------------------------------------------------
time:   00:35:22
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075801/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075801/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ef0b6f0e635041b984631e96617e23fc

    Command failed on smithi063 with status 1: 'sudo rm -rf --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp'

[7075807]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:24:11
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075807/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075807/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b896e667468a4a9689c3f7595ee6d7e1

    Command failed on smithi111 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    8977ca66-5183-11ed-8438-001a4aab830c -e
    sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\''

[7075821]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
-----------------------------------------------------------------
time:   00:11:45
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075821/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075821/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5c29580a7fdd4e67b0f677ad54e9bf2f

    Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)

[7075829]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
-----------------------------------------------------------------
time:   00:16:41
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075829/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075829/
sentry: https://sentry.ceph.com/organizations/ceph/?query=30e9df68d8f849e79e9395f3179fda53

    Test failure: test_dump_subtrees
    (tasks.cephfs.test_admin.TestAdminCommandDumpTree)

[7075831]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:11:30
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075831/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075831/
sentry: https://sentry.ceph.com/organizations/ceph/?query=85b66b23a53644629b1efed4fb33c70c

    Test failure: test_volume_create (tasks.cephfs.test_volumes.TestVolumes)

[7075841]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
-----------------------------------------------------------------
time:   00:14:35
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075841/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075841/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4e65733ce5ba429d8398a913ceb8f754

    Test failure: test_rebuild_simple
    (tasks.cephfs.test_recovery_pool.TestRecoveryPool)

[7075843]  fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}}
-----------------------------------------------------------------
time:   00:11:53
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075843/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075843/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fbc88673ad3a4787aaf86f83e9254e38

    Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps)

[7075845]  fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:11:58
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075845/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075845/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a625cd8c682443f8840910cb30e13e02

    Test failure: test_join_fs_drop
    (tasks.cephfs.test_failover.TestClusterAffinity)

[7075851]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree}
-----------------------------------------------------------------
time:   00:10:44
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075851/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075851/
sentry: https://sentry.ceph.com/organizations/ceph/?query=57f63de97106486682a006088b5a9888

    Test failure: test_basic (tasks.cephfs.test_dump_tree.TestDumpTree)

[7075853]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:18:51
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075853/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075853/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a83f1ad6e1a144c5845987c74270213f

    Command failed on smithi154 with status 100: 'sudo
    DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="
    --force-confdef" -o Dpkg::Options::="--force-confold" install bison flex
    libelf-dev libssl-dev network-manager iproute2 util-linux dump indent
    libaio-dev libtool-bin uuid-dev xfslibs-dev'

[7075857]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair}
-----------------------------------------------------------------
time:   00:13:26
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075857/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075857/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0d1c69a71b6941ec9b10be01ec6632aa

    Test failure: test_backtrace_repair
    (tasks.cephfs.test_auto_repair.TestMDSAutoRepair)

[7075859]  fs/upgrade/volumes/import-legacy/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04}
-----------------------------------------------------------------
time:   01:05:57
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075859/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075859/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4b804e62add54bcf836444dca08a0d7a

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun
    tu%2F18.04%2Fx86_64&sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a (Caused
    by NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7f8c9905cba8>: Failed to establish a new connection: [Errno 113] No route
    to host',))

[7075861]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:15:25
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075861/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075861/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dafdf80646514739994b800671f50d69

    Command failed on smithi006 with status 128: 'rm -rf
    /home/ubuntu/cephtest/clone.client.0 && git clone
    https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd
    /home/ubuntu/cephtest/clone.client.0 && git checkout
    d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a'

[7075865]  fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:10:17
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075865/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075865/
sentry: https://sentry.ceph.com/organizations/ceph/?query=941346e00e1b4e93af88cdae231046b9

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent
    os%2F8%2Fx86_64&sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a (Caused by
    NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7ffb576c0b38>: Failed to establish a new connection: [Errno 113] No route
    to host',))

[7075867]  fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb}
-----------------------------------------------------------------
time:   00:08:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075867/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075867/
sentry: https://sentry.ceph.com/organizations/ceph/?query=86329495b36540fe96791aba84f99059

    Command failed on smithi044 with status 1: 'sudo yum -y install ceph-
    radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-
    mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-
    mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2
    librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-
    mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-
    libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel
    libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-
    devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex
    elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux
    libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-
    linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel
    bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-
    linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump
    xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel'

[7075875]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075875/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075875/
sentry: https://sentry.ceph.com/organizations/ceph/?query=194ab509862b415ea29141985fb2002d

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun
    tu%2F20.04%2Fx86_64&ref=wip-yuri2-testing-2022-10-19-1252-pacific (Caused
    by NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7f610394fdd8>: Failed to establish a new connection: [Errno 113] No route
    to host',))

[7075877]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075877/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075877/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7e5b8a0f42de46708b238616a8c37da3

    Command failed on smithi188 with status 1: 'sudo yum install -y kernel'

[7075879]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075879/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075879/
sentry: https://sentry.ceph.com/organizations/ceph/?query=596b5764252f49a7abdbf56e5cd8fc4e

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent
    os%2F8%2Fx86_64&ref=wip-yuri2-testing-2022-10-19-1252-pacific (Caused by
    NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7f310fb9ce80>: Failed to establish a new connection: [Errno 113] No route
    to host',))

[7075881]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075881/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075881/
sentry: https://sentry.ceph.com/organizations/ceph/?query=13aa2c57104b47f9926664a45737772b

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent
    os%2F8%2Fx86_64&ref=wip-yuri2-testing-2022-10-19-1252-pacific (Caused by
    NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7efd29a96e80>: Failed to establish a new connection: [Errno 113] No route
    to host',))

[7075883]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075883/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075883/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f4ff61117f69435783fad095e8bb51de

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent
    os%2F8%2Fx86_64&ref=wip-yuri2-testing-2022-10-19-1252-pacific (Caused by
    NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7f2eac182dd8>: Failed to establish a new connection: [Errno -3] Temporary
    failure in name resolution',))

[7075885]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075885/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075885/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9ad3b0e365d14ee59599b6a15d1a36bb

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent
    os%2F8%2Fx86_64&ref=wip-yuri2-testing-2022-10-19-1252-pacific (Caused by
    NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7f7c1d611278>: Failed to establish a new connection: [Errno -3] Temporary
    failure in name resolution',))

[7075887]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075887/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075887/
sentry: https://sentry.ceph.com/organizations/ceph/?query=18607e3c094848b7b0d758274940ebe4

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent
    os%2F8%2Fx86_64&ref=wip-yuri2-testing-2022-10-19-1252-pacific (Caused by
    NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7f1de1e3ae10>: Failed to establish a new connection: [Errno -3] Temporary
    failure in name resolution',))

[7075889]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075889/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075889/
sentry: https://sentry.ceph.com/organizations/ceph/?query=34b78ff5dd7147d9ac0287eb5dd65adf

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=cent
    os%2F8%2Fx86_64&ref=wip-yuri2-testing-2022-10-19-1252-pacific (Caused by
    NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7f52f7116eb8>: Failed to establish a new connection: [Errno -3] Temporary
    failure in name resolution',))

[7075891]  fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075891/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075891/
sentry: https://sentry.ceph.com/organizations/ceph/?query=631769f9bc4c42759e7042d98e000fdc

    HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded
    with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubun
    tu%2F20.04%2Fx86_64&ref=wip-yuri2-testing-2022-10-19-1252-pacific (Caused
    by NewConnectionError('<urllib3.connection.HTTPSConnection object at
    0x7f6bec41ae80>: Failed to establish a new connection: [Errno -3] Temporary
    failure in name resolution',))

[7075897]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
-----------------------------------------------------------------
time:   00:18:22
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075897/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075897/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9cea4df6ecae4442813fe2b390c88e73

    Test failure: test_abort_conn
    (tasks.cephfs.test_client_recovery.TestClientRecovery)

[7075905]  fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:18:14
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075905/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075905/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b5445adcf4a246dda9b1aa984ee932b3

    Test failure: test_mount_all_caps_absent
    (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)

[7075913]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage}
-----------------------------------------------------------------
time:   00:16:03
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075913/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075913/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dfc8fc4fe44d4ef4bf487cfeea28c2cb

    Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage)

[7075917]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}
-----------------------------------------------------------------
time:   00:13:41
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075917/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075917/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9fa4ff1603534e54a5c993a2d2f73daa

    Test failure: test_fragmented_injection
    (tasks.cephfs.test_data_scan.TestDataScan)

[7075919]  fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
-----------------------------------------------------------------
time:   00:23:18
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075919/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075919/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d908b9b4f49a4a0fb09745a3af8bbb56

    Command failed on smithi074 with status 1: 'sudo
    /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c
    /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid
    6601f27a-518b-11ed-8438-001a4aab830c -e
    sha1=d88f759b6ee30a8b0d9b03e2bbb29f39d48f9f2a -- bash -c \'ceph versions |
    jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\''

[7075925]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports}
-----------------------------------------------------------------
time:   00:14:24
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075925/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075925/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b4286e278e4148d1baf3e9fd383972a0

    Test failure: test_ephemeral_pin_dist_conf_off
    (tasks.cephfs.test_exports.TestEphemeralPins)

[7075929]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
-----------------------------------------------------------------
time:   00:14:29
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075929/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075929/
sentry: https://sentry.ceph.com/organizations/ceph/?query=497c2e8e777d4f43954596343b4bf2a9

    Test failure: test_binary_metafile_on_legacy_to_v1_upgrade
    (tasks.cephfs.test_volumes.TestMisc)

[7075933]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}
-----------------------------------------------------------------
time:   00:16:43
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075933/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075933/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4b242ed829c54a30807655b23a419523

    Test failure: test_apply_tag
    (tasks.cephfs.test_forward_scrub.TestForwardScrub)

[7075937]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment}
-----------------------------------------------------------------
time:   00:15:21
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075937/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075937/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e93e964250604cb89c06a375a670b431

    Test failure: test_deep_split
    (tasks.cephfs.test_fragment.TestFragmentation)

[7075947]  fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:10:56
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075947/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075947/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f62698cdc32646338e4396427f3711d1

    Test failure: test_drop_cache_command
    (tasks.cephfs.test_misc.TestCacheDrop)

[7075953]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
-----------------------------------------------------------------
time:   00:10:58
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075953/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075953/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dbb50d5fbdc446f18e717d0053fc2a6b

    Test failure: test_inject_to_empty
    (tasks.cephfs.test_journal_repair.TestJournalRepair)



Dead
=================================================================
[7075710]  fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075710/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075710/

    hit max job timeout

[7075712]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075712/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075712/

    hit max job timeout

[7075714]  fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075714/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075714/

    hit max job timeout

[7075716]  fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075716/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075716/

    hit max job timeout

[7075718]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075718/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075718/

    hit max job timeout

[7075720]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075720/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075720/

    hit max job timeout

[7075722]  fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075722/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075722/

    hit max job timeout

[7075724]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075724/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075724/

    hit max job timeout

[7075726]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075726/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075726/

    hit max job timeout

[7075729]  fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075729/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075729/

    hit max job timeout

[7075730]  fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075730/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075730/

    hit max job timeout

[7075732]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075732/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075732/

    hit max job timeout

[7075734]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075734/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075734/

    hit max job timeout

[7075736]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075736/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075736/

    hit max job timeout

[7075738]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075738/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075738/

    hit max job timeout

[7075855]  fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep}
-----------------------------------------------------------------
time:   00:10:55
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075855/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075855/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eb49ed393974443ebc22220f2aacdd4a

    {'smithi110.front.sepia.ceph.com': {'changed': True, 'msg': 'All items
    completed', 'results': [{'_ansible_item_label': 'ceph/python3-asyncssh',
    '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True,
    'cmd': ['dnf', '-y', 'copr', 'enable', 'ceph/python3-asyncssh'], 'delta':
    '0:00:15.346051', 'end': '2022-10-21 21:31:10.601574', 'failed': True,
    'invocation': {'module_args': {'_raw_params': 'dnf -y copr enable
    ceph/python3-asyncssh', '_uses_shell': False, 'argv': None, 'chdir': None,
    'creates': None, 'executable': None, 'removes': None, 'stdin': None,
    'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}},
    'item': 'ceph/python3-asyncssh', 'msg': 'non-zero return code', 'rc': 1,
    'start': '2022-10-21 21:30:55.255523', 'stderr': 'Enabling a Copr
    repository. Please note that this repository is not part\nof the main
    distribution, and quality may vary.\n\nThe Fedora Project does not exercise
    any power over the contents of\nthis repository beyond the rules outlined
    in the Copr FAQ
    at\n<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-
    build-in-copr>,\nand packages are not held to any quality or security
    level.\n\nPlease do not file bug reports about these packages in
    Fedora\nBugzilla. In case of problems, contact the owner of this
    repository.\nError: Failed to connect to https://copr.fedorainfracloud.org/
    coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Name or
    service not known', 'stderr_lines': ['Enabling a Copr repository. Please
    note that this repository is not part', 'of the main distribution, and
    quality may vary.', '', 'The Fedora Project does not exercise any power
    over the contents of', 'this repository beyond the rules outlined in the
    Copr FAQ at',
    '<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-
    build-in-copr>,', 'and packages are not held to any quality or security
    level.', '', 'Please do not file bug reports about these packages in
    Fedora', 'Bugzilla. In case of problems, contact the owner of this
    repository.', 'Error: Failed to connect to https://copr.fedorainfracloud.or
    g/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Name or
    service not known'], 'stdout': '', 'stdout_lines': []}], 'warnings':
    ["Consider using the dnf module rather than running 'dnf'.  If you need to
    use command because dnf is insufficient you can add 'warn: false' to this
    command task or set 'command_warnings=False' in ansible.cfg to get rid of
    this message."]}, 'smithi175.front.sepia.ceph.com': {'_ansible_no_log':
    False, 'changed': False, 'cmd': ['/bin/pip3', 'install', '-U',
    'git+https://github.com/sebastian-philipp/registries-conf-ctl'],
    'invocation': {'module_args': {'chdir': None, 'editable': False,
    'executable': 'pip3', 'extra_args': None, 'name':
    ['git+https://github.com/sebastian-philipp/registries-conf-ctl'],
    'requirements': None, 'state': 'latest', 'umask': None, 'version': None,
    'virtualenv': None, 'virtualenv_command': 'virtualenv',
    'virtualenv_python': None, 'virtualenv_site_packages': False}}, 'msg':
    'stdout: Collecting git+https://github.com/sebastian-philipp/registries-
    conf-ctl\n  Cloning https://github.com/sebastian-philipp/registries-conf-
    ctl to ./pip-_8z5oo5m-build\nfatal: unable to access
    \'https://github.com/sebastian-philipp/registries-conf-ctl/\': Could not
    resolve host: github.com\n\n:stderr: WARNING: Running pip install with root
    privileges is generally not a good idea. Try `pip3 install --user`
    instead.\nCommand "git clone -q https://github.com/sebastian-
    philipp/registries-conf-ctl /tmp/pip-_8z5oo5m-build" failed with error code
    128 in None\n'}}

[7075863]  fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}}
-----------------------------------------------------------------
time:   00:08:03
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075863/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075863/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9277c3050ca9455297cc46e328f4455d

    {'smithi045.front.sepia.ceph.com': {'_ansible_no_log': False,
    'cache_update_time': 1666387794, 'cache_updated': False, 'changed': False,
    'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean':
    False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None,
    'default_release': None, 'dpkg_options': 'force-confdef,force-confold',
    'force': True, 'force_apt_get': False, 'install_recommends': None, 'name':
    ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io',
    'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj',
    'python3-gevent', 'python3-numpy', 'python3-matplotlib',
    'python3-setuptools', 'python-dev', 'libfcgi0ldbl', 'libev-dev', 'perl',
    'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb',
    'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1',
    'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind',
    'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc',
    'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin',
    'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage',
    'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-
    iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server',
    'software-properties-common', 'libgnutls30'], 'only_upgrade': False,
    'package': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin',
    'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv',
    'python3-configobj', 'python3-gevent', 'python3-numpy',
    'python3-matplotlib', 'python3-setuptools', 'python-dev', 'libfcgi0ldbl',
    'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential',
    'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk',
    'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench',
    'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-
    dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev',
    'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh',
    'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-
    jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common',
    'nfs-kernel-server', 'software-properties-common', 'libgnutls30'],
    'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache':
    None, 'update_cache_retries': 5, 'update_cache_retry_max_delay': 12,
    'upgrade': None}}, 'msg': '\'/usr/bin/apt-get -y -o
    "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
    --force-yes   install \'qemu-system-x86\' \'git-core\'\' failed: W:
    --force-yes is deprecated, use one of the options starting with --allow
    instead.\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/
    git/git_2.25.1-1ubuntu3.6_amd64.deb  Could not connect to
    archive.ubuntu.com:80 (91.189.91.38). - connect (113: No route to host)
    Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect
    (113: No route to host) Could not connect to archive.ubuntu.com:80
    (185.125.190.36). - connect (113: No route to host) Could not connect to
    archive.ubuntu.com:80 (91.189.91.39). - connect (113: No route to host)
    Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::19), connection
    timed out Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::16),
    connection timed out Could not connect to archive.ubuntu.com:80
    (2001:67c:1562::18), connection timed out Could not connect to
    archive.ubuntu.com:80 (2001:67c:1562::15), connection timed out\nE: Failed
    to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librados2_15.2.1
    6-0ubuntu0.20.04.1_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:\nE: Failed to fetch http://archive.ubuntu.com/ubun
    tu/pool/main/c/ceph/librbd1_15.2.16-0ubuntu0.20.04.1_amd64.deb  Unable to
    connect to archive.ubuntu.com:http:\nE: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-block-
    extra_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:\nE: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-system-
    common_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:\nE: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-
    system-x86_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:\nE: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-
    utils_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:\nE: Unable to fetch some archives, maybe run apt-
    get update or try with --fix-missing?\n', 'rc': 100, 'stderr': 'W: --force-
    yes is deprecated, use one of the options starting with --allow
    instead.\nE: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/
    git/git_2.25.1-1ubuntu3.6_amd64.deb  Could not connect to
    archive.ubuntu.com:80 (91.189.91.38). - connect (113: No route to host)
    Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect
    (113: No route to host) Could not connect to archive.ubuntu.com:80
    (185.125.190.36). - connect (113: No route to host) Could not connect to
    archive.ubuntu.com:80 (91.189.91.39). - connect (113: No route to host)
    Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::19), connection
    timed out Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::16),
    connection timed out Could not connect to archive.ubuntu.com:80
    (2001:67c:1562::18), connection timed out Could not connect to
    archive.ubuntu.com:80 (2001:67c:1562::15), connection timed out\nE: Failed
    to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librados2_15.2.1
    6-0ubuntu0.20.04.1_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:\nE: Failed to fetch http://archive.ubuntu.com/ubun
    tu/pool/main/c/ceph/librbd1_15.2.16-0ubuntu0.20.04.1_amd64.deb  Unable to
    connect to archive.ubuntu.com:http:\nE: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-block-
    extra_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:\nE: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-system-
    common_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:\nE: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-
    system-x86_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:\nE: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-
    utils_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:\nE: Unable to fetch some archives, maybe run apt-
    get update or try with --fix-missing?\n', 'stderr_lines': ['W: --force-yes
    is deprecated, use one of the options starting with --allow instead.', 'E:
    Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/g/git/git_2.25.
    1-1ubuntu3.6_amd64.deb  Could not connect to archive.ubuntu.com:80
    (91.189.91.38). - connect (113: No route to host) Could not connect to
    archive.ubuntu.com:80 (185.125.190.39). - connect (113: No route to host)
    Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect
    (113: No route to host) Could not connect to archive.ubuntu.com:80
    (91.189.91.39). - connect (113: No route to host) Could not connect to
    archive.ubuntu.com:80 (2620:2d:4000:1::19), connection timed out Could not
    connect to archive.ubuntu.com:80 (2620:2d:4000:1::16), connection timed out
    Could not connect to archive.ubuntu.com:80 (2001:67c:1562::18), connection
    timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::15),
    connection timed out', 'E: Failed to fetch http://archive.ubuntu.com/ubuntu
    /pool/main/c/ceph/librados2_15.2.16-0ubuntu0.20.04.1_amd64.deb  Unable to
    connect to archive.ubuntu.com:http:', 'E: Failed to fetch http://archive.ub
    untu.com/ubuntu/pool/main/c/ceph/librbd1_15.2.16-0ubuntu0.20.04.1_amd64.deb
    Unable to connect to archive.ubuntu.com:http:', 'E: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-block-
    extra_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:', 'E: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-system-
    common_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:', 'E: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-
    system-x86_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:', 'E: Failed to fetch
    http://security.ubuntu.com/ubuntu/pool/main/q/qemu/qemu-
    utils_4.2-3ubuntu6.23_amd64.deb  Unable to connect to
    archive.ubuntu.com:http:', 'E: Unable to fetch some archives, maybe run
    apt-get update or try with --fix-missing?'], 'stdout': 'Reading package
    lists...\nBuilding dependency tree...\nReading state information...\nThe
    following additional packages will be installed:\n  librados2 librbd1 qemu-
    block-extra qemu-system-common qemu-utils\nSuggested packages:\n  git-
    daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n
    gitweb git-cvs git-mediawiki git-svn samba vde2 debootstrap\nThe following
    NEW packages will be installed:\n  librados2 librbd1 qemu-block-extra qemu-
    system-common qemu-system-x86\n  qemu-utils\nThe following packages will be
    upgraded:\n  git\n1 upgraded, 6 newly installed, 0 to remove and 13 not
    upgraded.\nNeed to get 18.2 MB of archives.\nAfter this operation, 62.7 MB
    of additional disk space will be used.\nIgn:1
    http://archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64
    1:2.25.1-1ubuntu3.6\nErr:2 http://archive.ubuntu.com/ubuntu focal-
    updates/main amd64 librados2 amd64 15.2.16-0ubuntu0.20.04.1\n  Unable to
    connect to archive.ubuntu.com:http:\nErr:3 http://archive.ubuntu.com/ubuntu
    focal-updates/main amd64 librbd1 amd64 15.2.16-0ubuntu0.20.04.1\n  Unable
    to connect to archive.ubuntu.com:http:\nIgn:4
    http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra
    amd64 1:4.2-3ubuntu6.23\nIgn:5 http://archive.ubuntu.com/ubuntu focal-
    updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.23\nIgn:6
    http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86
    amd64 1:4.2-3ubuntu6.23\nIgn:7 http://archive.ubuntu.com/ubuntu focal-
    updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.23\nErr:1
    http://security.ubuntu.com/ubuntu focal-updates/main amd64 git amd64
    1:2.25.1-1ubuntu3.6\n  Could not connect to archive.ubuntu.com:80
    (91.189.91.38). - connect (113: No route to host) Could not connect to
    archive.ubuntu.com:80 (185.125.190.39). - connect (113: No route to host)
    Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect
    (113: No route to host) Could not connect to archive.ubuntu.com:80
    (91.189.91.39). - connect (113: No route to host) Could not connect to
    archive.ubuntu.com:80 (2620:2d:4000:1::19), connection timed out Could not
    connect to archive.ubuntu.com:80 (2620:2d:4000:1::16), connection timed out
    Could not connect to archive.ubuntu.com:80 (2001:67c:1562::18), connection
    timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::15),
    connection timed out\nErr:4 http://security.ubuntu.com/ubuntu focal-
    updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.23\n  Unable to
    connect to archive.ubuntu.com:http:\nErr:5
    http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-
    common amd64 1:4.2-3ubuntu6.23\n  Unable to connect to
    archive.ubuntu.com:http:\nErr:6 http://security.ubuntu.com/ubuntu focal-
    updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.23\n  Unable to
    connect to archive.ubuntu.com:http:\nErr:7
    http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64
    1:4.2-3ubuntu6.23\n  Unable to connect to archive.ubuntu.com:http:\n',
    'stdout_lines': ['Reading package lists...', 'Building dependency tree...',
    'Reading state information...', 'The following additional packages will be
    installed:', '  librados2 librbd1 qemu-block-extra qemu-system-common qemu-
    utils', 'Suggested packages:', '  git-daemon-run | git-daemon-sysvinit git-
    doc git-el git-email git-gui gitk', '  gitweb git-cvs git-mediawiki git-svn
    samba vde2 debootstrap', 'The following NEW packages will be installed:', '
    librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86', '
    qemu-utils', 'The following packages will be upgraded:', '  git', '1
    upgraded, 6 newly installed, 0 to remove and 13 not upgraded.', 'Need to
    get 18.2 MB of archives.', 'After this operation, 62.7 MB of additional
    disk space will be used.', 'Ign:1 http://archive.ubuntu.com/ubuntu focal-
    updates/main amd64 git amd64 1:2.25.1-1ubuntu3.6', 'Err:2
    http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64
    15.2.16-0ubuntu0.20.04.1', '  Unable to connect to
    archive.ubuntu.com:http:', 'Err:3 http://archive.ubuntu.com/ubuntu focal-
    updates/main amd64 librbd1 amd64 15.2.16-0ubuntu0.20.04.1', '  Unable to
    connect to archive.ubuntu.com:http:', 'Ign:4
    http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra
    amd64 1:4.2-3ubuntu6.23', 'Ign:5 http://archive.ubuntu.com/ubuntu focal-
    updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.23', 'Ign:6
    http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86
    amd64 1:4.2-3ubuntu6.23', 'Ign:7 http://archive.ubuntu.com/ubuntu focal-
    updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.23', 'Err:1
    http://security.ubuntu.com/ubuntu focal-updates/main amd64 git amd64
    1:2.25.1-1ubuntu3.6', '  Could not connect to archive.ubuntu.com:80
    (91.189.91.38). - connect (113: No route to host) Could not connect to
    archive.ubuntu.com:80 (185.125.190.39). - connect (113: No route to host)
    Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect
    (113: No route to host) Could not connect to archive.ubuntu.com:80
    (91.189.91.39). - connect (113: No route to host) Could not connect to
    archive.ubuntu.com:80 (2620:2d:4000:1::19), connection timed out Could not
    connect to archive.ubuntu.com:80 (2620:2d:4000:1::16), connection timed out
    Could not connect to archive.ubuntu.com:80 (2001:67c:1562::18), connection
    timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1562::15),
    connection timed out', 'Err:4 http://security.ubuntu.com/ubuntu focal-
    updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.23', '  Unable to
    connect to archive.ubuntu.com:http:', 'Err:5
    http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-
    common amd64 1:4.2-3ubuntu6.23', '  Unable to connect to
    archive.ubuntu.com:http:', 'Err:6 http://security.ubuntu.com/ubuntu focal-
    updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.23', '  Unable to
    connect to archive.ubuntu.com:http:', 'Err:7
    http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64
    1:4.2-3ubuntu6.23', '  Unable to connect to archive.ubuntu.com:http:']},
    'smithi137.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'cmd': ['/usr/bin/pip3', 'install', '-U',
    'git+https://github.com/sebastian-philipp/registries-conf-ctl'],
    'invocation': {'module_args': {'chdir': None, 'editable': False,
    'executable': 'pip3', 'extra_args': None, 'name':
    ['git+https://github.com/sebastian-philipp/registries-conf-ctl'],
    'requirements': None, 'state': 'latest', 'umask': None, 'version': None,
    'virtualenv': None, 'virtualenv_command': 'virtualenv',
    'virtualenv_python': None, 'virtualenv_site_packages': False}}, 'msg':
    "stdout: Collecting git+https://github.com/sebastian-philipp/registries-
    conf-ctl\n  Cloning https://github.com/sebastian-philipp/registries-conf-
    ctl to ./pip-req-build-jb9nktur\n\n:stderr:   Running command git clone -q
    https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-
    build-jb9nktur\n  fatal: unable to access 'https://github.com/sebastian-
    philipp/registries-conf-ctl/': Could not resolve host: github.com\nERROR:
    Command errored out with exit status 128: git clone -q
    https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-
    build-jb9nktur Check the logs for full command output.\n"},
    'smithi057.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed':
    False, 'cmd': ['/usr/bin/pip3', 'install', '-U',
    'git+https://github.com/sebastian-philipp/registries-conf-ctl'],
    'invocation': {'module_args': {'chdir': None, 'editable': False,
    'executable': 'pip3', 'extra_args': None, 'name':
    ['git+https://github.com/sebastian-philipp/registries-conf-ctl'],
    'requirements': None, 'state': 'latest', 'umask': None, 'version': None,
    'virtualenv': None, 'virtualenv_command': 'virtualenv',
    'virtualenv_python': None, 'virtualenv_site_packages': False}}, 'msg':
    "stdout: Collecting git+https://github.com/sebastian-philipp/registries-
    conf-ctl\n  Cloning https://github.com/sebastian-philipp/registries-conf-
    ctl to ./pip-req-build-d156ofl8\n\n:stderr:   Running command git clone -q
    https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-
    build-d156ofl8\n  fatal: unable to access 'https://github.com/sebastian-
    philipp/registries-conf-ctl/': Could not resolve host: github.com\nERROR:
    Command errored out with exit status 128: git clone -q
    https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-
    build-d156ofl8 Check the logs for full command output.\n"}}

[7075869]  fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:03:45
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075869/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075869/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8731bca4e5d24e33b655679bbb5496f1

    {'smithi083.front.sepia.ceph.com': {'_ansible_delegated_vars':
    {'ansible_connection': 'local', 'ansible_host': 'localhost',
    'ansible_port': None, 'ansible_user': 'cm'}, '_ansible_no_log': False,
    'attempts': 5, 'changed': False, 'cmd': ['/usr/bin/git', 'fetch', '--tags',
    '--force', 'origin'], 'invocation': {'module_args': {'accept_hostkey':
    False, 'archive': None, 'archive_prefix': None, 'bare': False, 'clone':
    True, 'depth': None, 'dest': '/home/teuthworker/.cache/src/keys',
    'executable': None, 'force': True, 'gpg_whitelist': [], 'key_file': None,
    'recursive': True, 'reference': None, 'refspec': None, 'remote': 'origin',
    'repo': 'https://github.com/ceph/keys', 'separate_git_dir': None,
    'ssh_opts': None, 'track_submodules': False, 'umask': None, 'update': True,
    'verify_commit': False, 'version': 'main'}}, 'msg': "Failed to download
    remote objects and refs:  fatal: unable to access
    'https://github.com/ceph/keys/': Could not resolve host: github.com\n"},
    'smithi049.front.sepia.ceph.com': {'_ansible_delegated_vars':
    {'ansible_connection': 'local', 'ansible_host': 'localhost',
    'ansible_port': None, 'ansible_user': 'cm'}, '_ansible_no_log': False,
    'attempts': 5, 'changed': False, 'cmd': ['/usr/bin/git', 'fetch', '--tags',
    '--force', 'origin'], 'invocation': {'module_args': {'accept_hostkey':
    False, 'archive': None, 'archive_prefix': None, 'bare': False, 'clone':
    True, 'depth': None, 'dest': '/home/teuthworker/.cache/src/keys',
    'executable': None, 'force': True, 'gpg_whitelist': [], 'key_file': None,
    'recursive': True, 'reference': None, 'refspec': None, 'remote': 'origin',
    'repo': 'https://github.com/ceph/keys', 'separate_git_dir': None,
    'ssh_opts': None, 'track_submodules': False, 'umask': None, 'update': True,
    'verify_commit': False, 'version': 'main'}}, 'msg': "Failed to download
    remote objects and refs:  fatal: unable to access
    'https://github.com/ceph/keys/': Could not resolve host: github.com\n"}}

[7075871]  fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}
-----------------------------------------------------------------
time:   00:08:01
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075871/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075871/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5ee1a78bb0bc4af298187ef9d40a96d6

    {'smithi203.front.sepia.ceph.com': {'_ansible_delegated_vars':
    {'ansible_connection': 'local', 'ansible_host': 'localhost',
    'ansible_port': None, 'ansible_user': 'cm'}, '_ansible_no_log': False,
    'attempts': 5, 'changed': False, 'cmd': ['/usr/bin/git', 'fetch', '--tags',
    '--force', 'origin'], 'invocation': {'module_args': {'accept_hostkey':
    False, 'archive': None, 'archive_prefix': None, 'bare': False, 'clone':
    True, 'depth': None, 'dest': '/home/teuthworker/.cache/src/keys',
    'executable': None, 'force': True, 'gpg_whitelist': [], 'key_file': None,
    'recursive': True, 'reference': None, 'refspec': None, 'remote': 'origin',
    'repo': 'https://github.com/ceph/keys', 'separate_git_dir': None,
    'ssh_opts': None, 'track_submodules': False, 'umask': None, 'update': True,
    'verify_commit': False, 'version': 'main'}}, 'msg': "Failed to download
    remote objects and refs:  fatal: unable to access
    'https://github.com/ceph/keys/': Could not resolve host: github.com\n"},
    'smithi088.front.sepia.ceph.com': {'_ansible_delegated_vars':
    {'ansible_connection': 'local', 'ansible_host': 'localhost',
    'ansible_port': None, 'ansible_user': 'cm'}, '_ansible_no_log': False,
    'attempts': 5, 'changed': False, 'cmd': ['/usr/bin/git', 'fetch', '--tags',
    '--force', 'origin'], 'invocation': {'module_args': {'accept_hostkey':
    False, 'archive': None, 'archive_prefix': None, 'bare': False, 'clone':
    True, 'depth': None, 'dest': '/home/teuthworker/.cache/src/keys',
    'executable': None, 'force': True, 'gpg_whitelist': [], 'key_file': None,
    'recursive': True, 'reference': None, 'refspec': None, 'remote': 'origin',
    'repo': 'https://github.com/ceph/keys', 'separate_git_dir': None,
    'ssh_opts': None, 'track_submodules': False, 'umask': None, 'update': True,
    'verify_commit': False, 'version': 'main'}}, 'msg': "Failed to download
    remote objects and refs:  fatal: unable to access
    'https://github.com/ceph/keys/': Could not resolve host: github.com\n"}}

[7075873]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
-----------------------------------------------------------------
time:   00:13:26
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075873/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075873/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1d1e6f85039e4d45be41271cd19baa52

    {'smithi085.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts':
    24, 'changed': False, 'invocation': {'module_args': {'comment': None,
    'exclusive': False, 'follow': False, 'key':
    'https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub',
    'key_options': None, 'manage_dir': True, 'path': None, 'state': 'present',
    'user': 'ubuntu', 'validate_certs': True}}, 'msg': 'Error getting key from:
    https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub'},
    'smithi017.front.sepia.ceph.com': {'_ansible_no_log': False, 'attempts':
    24, 'changed': False, 'invocation': {'module_args': {'comment': None,
    'exclusive': False, 'follow': False, 'key':
    'https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub',
    'key_options': None, 'manage_dir': True, 'path': None, 'state': 'present',
    'user': 'ubuntu', 'validate_certs': True}}, 'msg': 'Error getting key from:
    https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub'}}



Running
=================================================================
[7075923] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075923/



Pass
=================================================================
[7075626] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}}
time:   00:23:10
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075626/

[7075628] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}}
time:   00:22:35
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075628/

[7075632] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   00:28:11
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075632/

[7075636] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:43:29
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075636/

[7075638] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:17:54
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075638/

[7075642] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:13:48
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075642/

[7075644] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:53:59
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075644/

[7075646] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:57:17
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075646/

[7075648] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind}
time:   01:52:30
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075648/

[7075652] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}}
time:   00:36:33
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075652/

[7075654] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:10:47
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075654/

[7075661] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   00:39:03
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075661/

[7075662] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:13:02
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075662/

[7075670] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:23:28
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075670/

[7075675] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}}
time:   00:18:20
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075675/

[7075677] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry}
time:   00:15:34
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075677/

[7075679] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:13:29
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075679/

[7075682] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:06
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075682/

[7075684] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   00:11:20
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075684/

[7075686] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:13:37
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075686/

[7075690] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:20:21
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075690/

[7075692] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:16:59
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075692/

[7075694] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}}
time:   00:13:59
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075694/

[7075698] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:27:34
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075698/

[7075704] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:16:26
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075704/

[7075743] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   11:07:12
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075743/

[7075749] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   05:19:55
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075749/

[7075753] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}}
time:   05:37:14
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075753/

[7075755] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   05:25:00
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075755/

[7075759] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   05:17:10
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075759/

[7075761] fs/bugs/client_trim_caps/{begin centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137}
time:   05:09:26
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075761/

[7075765] fs/full/{begin clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full}
time:   05:15:36
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075765/

[7075767] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client}
time:   05:09:43
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075767/

[7075771] fs/mirror-ha/{begin cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit}
time:   07:23:14
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075771/

[7075773] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
time:   05:16:55
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075773/

[7075779] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   04:08:27
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075779/

[7075783] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   04:01:01
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075783/

[7075789] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   02:50:33
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075789/

[7075795] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   01:49:22
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075795/

[7075799] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind}
time:   00:54:51
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075799/

[7075803] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay}
time:   00:18:12
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075803/

[7075805] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:44:53
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075805/

[7075809] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:18:11
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075809/

[7075811] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:19:29
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075811/

[7075813] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
time:   00:14:04
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075813/

[7075815] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}}
time:   01:20:53
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075815/

[7075817] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:15:29
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075817/

[7075819] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}}
time:   00:11:29
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075819/

[7075823] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:22
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075823/

[7075825] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:07
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075825/

[7075827] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:23:22
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075827/

[7075833] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:56:48
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075833/

[7075835] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
time:   00:11:11
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075835/

[7075837] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:18:58
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075837/

[7075839] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}}
time:   00:59:22
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075839/

[7075847] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:01:19
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075847/

[7075849] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   01:01:33
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075849/

[7075893] fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
time:   00:21:53
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075893/

[7075895] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:20:57
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075895/

[7075899] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:42:37
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075899/

[7075901] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
time:   00:28:46
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075901/

[7075903] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:48
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075903/

[7075907] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:17:04
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075907/

[7075909] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:52:22
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075909/

[7075911] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc}
time:   00:56:50
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075911/

[7075915] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
time:   01:27:26
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075915/

[7075921] fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}
time:   00:57:28
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075921/

[7075927] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}}
time:   00:19:38
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075927/

[7075931] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:22:26
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075931/

[7075935] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}}
time:   00:24:25
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075935/

[7075939] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:13:24
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075939/

[7075941] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:47
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075941/

[7075943] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
time:   00:16:24
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075943/

[7075945] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:40:37
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075945/

[7075949] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:16:22
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075949/

[7075951] fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}}
time:   00:25:55
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075951/

[7075955] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:18:09
info:   http://pulpito.ceph.com/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075955/