2023-06-22T17:48:39.876 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-22T17:48:39.877 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} --verbose --archive /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857 --name yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857/orig.config.yaml 2023-06-22T17:48:39.922 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T17:48:40.019 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857 branch: wip-yuri4-testing-2023-06-21-0846-quincy description: fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} email: yweinste@redhat.com first_in_suite: false job_id: '7311857' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-06-21-0846-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-06-21-0846-quincy sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 9037 sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2023-06-21-0846-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c targets: smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE8qRhuRUXj1iFiQd/Gr7ahxAb/xrAR5aYNjNZNzG8lND1ManSzV5hIdYStVClXN02MXcaxubtpjWEjQm1azrps= smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOlY0LD4+AaVWVxqWRbI1ZOcEjeJnJyA/uEqjVQqVqKkgaSUHAk8XiDdozZLzJuZTuX+5Sc911PDBBIpIciEguKTBN1jMRTf4XX0SDPQbjZfukPSSEBZGxWzuVyfPk1UlCGdlHbkGHH0xwvWBerXFjA7qSH4rXOR7NizG70cnbCdih0S/hzVNGJDLVSvJ5gAiqE6PxjQ7Zf3zLNiwzvMhD4dolr7HwhSnAOddGaqSo8ZU+6VM+BYWFy2syLN7tDV1f8UCYt26ItcV6MMvwQRI4usWpNssQnI8J2j2Dn2PmPBdNFnmwFp4UyKK7Aq0lT032ygSvVdD7Oa5ZKlFg1UaZcNXfAZudG/qXwN0rzebd4cvKVq0Hw3jpHiTLqYGqQtE2Cz1iFmmoD7LSoVBXGItMBZJw+L+UhlkM0+wXAk/44WSV0CRMOcR8aDx5NiCnHv5Vw3NED+SpkjGsLzFTwwWQQoWIf39TilTSGqpZrWnp9ulvrgIv7VAVEAD6fiaVPqU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 076bbebc42a14f7d568aaa78eabb0038327bcb23 timestamp: 2023-06-22_15:25:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-06-22T17:48:40.020 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa; will attempt to use it 2023-06-22T17:48:40.020 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks 2023-06-22T17:48:40.021 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-06-22T17:48:40.022 INFO:teuthology.task.internal:Checking packages... 2023-06-22T17:48:40.051 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '42ddc46e6e6582d8e29a5a982a81886fe028be2c' 2023-06-22T17:48:40.052 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-06-22T17:48:40.052 INFO:teuthology.packaging:ref: None 2023-06-22T17:48:40.052 INFO:teuthology.packaging:tag: None 2023-06-22T17:48:40.053 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-22T17:48:40.053 INFO:teuthology.packaging:sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T17:48:40.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-22T17:48:40.197 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-719.g42ddc46e 2023-06-22T17:48:40.198 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-06-22T17:48:40.207 INFO:teuthology.task.internal:no buildpackages task found 2023-06-22T17:48:40.208 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-06-22T17:48:40.242 INFO:teuthology.task.internal:Saving configuration 2023-06-22T17:48:40.262 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-06-22T17:48:40.268 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T17:48:40.296 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 17:43:43.671000', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE8qRhuRUXj1iFiQd/Gr7ahxAb/xrAR5aYNjNZNzG8lND1ManSzV5hIdYStVClXN02MXcaxubtpjWEjQm1azrps='} 2023-06-22T17:48:40.319 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 17:43:43.672957', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQqzXtf+Et+0f9kgVac0G9UygAMBJDTuZ4Fi6HFKHRurkB3TMNupTqoeHWwEw/6YTKRNGLxgHhCRxSYTH2Rhbw='} 2023-06-22T17:48:40.319 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-06-22T17:48:40.325 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-06-22T17:48:40.325 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-06-22T17:48:40.325 INFO:teuthology.run_tasks:Running task console_log... 2023-06-22T17:48:40.434 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd4317d7040>, signals=[15]) 2023-06-22T17:48:40.435 INFO:teuthology.run_tasks:Running task internal.connect... 2023-06-22T17:48:40.441 INFO:teuthology.task.internal:Opening connections... 2023-06-22T17:48:40.442 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2023-06-22T17:48:40.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:48:40.512 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2023-06-22T17:48:40.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:48:40.583 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-06-22T17:48:40.590 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2023-06-22T17:48:40.607 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2023-06-22T17:48:40.607 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2023-06-22T17:48:40.663 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Red Hat Enterprise Linux" 2023-06-22T17:48:40.663 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="8.4 (Ootpa)" 2023-06-22T17:48:40.663 INFO:teuthology.orchestra.run.smithi110.stdout:ID="rhel" 2023-06-22T17:48:40.664 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="fedora" 2023-06-22T17:48:40.664 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="8.4" 2023-06-22T17:48:40.664 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el8" 2023-06-22T17:48:40.664 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-22T17:48:40.664 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2023-06-22T17:48:40.664 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-22T17:48:40.665 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.redhat.com/" 2023-06-22T17:48:40.665 INFO:teuthology.orchestra.run.smithi110.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-22T17:48:40.665 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-22T17:48:40.665 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T17:48:40.665 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-22T17:48:40.665 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-22T17:48:40.665 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-22T17:48:40.666 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-22T17:48:40.666 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2023-06-22T17:48:40.692 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2023-06-22T17:48:40.709 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2023-06-22T17:48:40.709 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2023-06-22T17:48:40.765 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="Red Hat Enterprise Linux" 2023-06-22T17:48:40.766 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="8.4 (Ootpa)" 2023-06-22T17:48:40.766 INFO:teuthology.orchestra.run.smithi116.stdout:ID="rhel" 2023-06-22T17:48:40.766 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="fedora" 2023-06-22T17:48:40.766 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="8.4" 2023-06-22T17:48:40.766 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el8" 2023-06-22T17:48:40.767 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-22T17:48:40.767 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2023-06-22T17:48:40.767 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-22T17:48:40.767 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://www.redhat.com/" 2023-06-22T17:48:40.768 INFO:teuthology.orchestra.run.smithi116.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-22T17:48:40.768 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-22T17:48:40.768 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T17:48:40.768 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-22T17:48:40.769 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-22T17:48:40.769 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-22T17:48:40.769 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-22T17:48:40.769 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2023-06-22T17:48:40.791 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-06-22T17:48:40.801 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-06-22T17:48:40.859 INFO:teuthology.task.internal:Checking for old test directory... 2023-06-22T17:48:40.860 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2023-06-22T17:48:40.863 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2023-06-22T17:48:40.879 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-06-22T17:48:40.884 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-06-22T17:48:40.885 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2023-06-22T17:48:40.920 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2023-06-22T17:48:40.933 INFO:teuthology.orchestra.run.smithi110.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-22T17:48:40.936 INFO:teuthology.orchestra.run.smithi116.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-22T17:48:40.937 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-06-22T17:48:40.997 INFO:teuthology.run_tasks:Running task kernel... 2023-06-22T17:48:41.012 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-06-22T17:48:41.013 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-06-22T17:48:41.014 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-22T17:48:41.015 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-22T17:48:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:48:41.030 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2023-06-22T17:48:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:48:41.032 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-06-22T17:48:41.087 INFO:teuthology.orchestra.run.smithi110.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-22T17:48:41.088 INFO:teuthology.task.kernel:Running kernel on smithi110: 4.18.0-305.25.1.el8_4.x86_64 2023-06-22T17:48:41.088 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2023-06-22T17:48:41.090 INFO:teuthology.orchestra.run.smithi116.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-22T17:48:41.090 INFO:teuthology.task.kernel:Running kernel on smithi116: 4.18.0-305.25.1.el8_4.x86_64 2023-06-22T17:48:41.091 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2023-06-22T17:48:42.502 INFO:teuthology.orchestra.run.smithi116.stdout:Updating Subscription Management repositories. 2023-06-22T17:48:42.503 INFO:teuthology.orchestra.run.smithi116.stdout:Unable to read consumer identity 2023-06-22T17:48:42.647 INFO:teuthology.orchestra.run.smithi110.stdout:Updating Subscription Management repositories. 2023-06-22T17:48:42.648 INFO:teuthology.orchestra.run.smithi110.stdout:Unable to read consumer identity 2023-06-22T17:48:45.794 INFO:teuthology.orchestra.run.smithi110.stdout:Copr repo for python3-asyncssh owned by ceph 1.9 kB/s | 3.5 kB 00:01 2023-06-22T17:48:46.319 INFO:teuthology.orchestra.run.smithi116.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 3.5 kB 00:02 2023-06-22T17:48:46.994 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 24 MB/s | 16 MB 00:00 2023-06-22T17:48:47.598 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 16 MB 00:01 2023-06-22T17:48:50.177 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-06-22T17:48:50.838 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2023-06-22T17:48:50.861 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-22T17:48:50.862 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-22T17:48:50.862 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-22T17:48:51.018 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-06-22T17:48:51.019 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2023-06-22T17:48:51.020 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-06-22T17:48:51.131 DEBUG:teuthology.orchestra.run.smithi116:> echo no | sudo yum reinstall kernel || true 2023-06-22T17:48:51.475 INFO:teuthology.orchestra.run.smithi116.stdout:Updating Subscription Management repositories. 2023-06-22T17:48:51.476 INFO:teuthology.orchestra.run.smithi116.stdout:Unable to read consumer identity 2023-06-22T17:48:51.605 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-22T17:48:51.605 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-22T17:48:51.606 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-22T17:48:51.755 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-06-22T17:48:51.756 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2023-06-22T17:48:51.757 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-06-22T17:48:51.765 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:01 ago on Thu 22 Jun 2023 05:48:50 PM UTC. 2023-06-22T17:48:51.899 INFO:teuthology.orchestra.run.smithi116.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-22T17:48:51.899 INFO:teuthology.orchestra.run.smithi116.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:48:51.899 INFO:teuthology.orchestra.run.smithi116.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:48:51.913 DEBUG:teuthology.orchestra.run.smithi110:> echo no | sudo yum reinstall kernel || true 2023-06-22T17:48:51.928 INFO:teuthology.orchestra.run.smithi116.stderr:Error: No packages marked for reinstall. 2023-06-22T17:48:51.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum reinstall -y kernel || true 2023-06-22T17:48:52.255 INFO:teuthology.orchestra.run.smithi110.stdout:Updating Subscription Management repositories. 2023-06-22T17:48:52.256 INFO:teuthology.orchestra.run.smithi110.stdout:Unable to read consumer identity 2023-06-22T17:48:52.325 INFO:teuthology.orchestra.run.smithi116.stdout:Updating Subscription Management repositories. 2023-06-22T17:48:52.325 INFO:teuthology.orchestra.run.smithi116.stdout:Unable to read consumer identity 2023-06-22T17:48:52.543 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:02 ago on Thu 22 Jun 2023 05:48:50 PM UTC. 2023-06-22T17:48:52.609 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:02 ago on Thu 22 Jun 2023 05:48:50 PM UTC. 2023-06-22T17:48:52.672 INFO:teuthology.orchestra.run.smithi110.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-22T17:48:52.672 INFO:teuthology.orchestra.run.smithi110.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:48:52.673 INFO:teuthology.orchestra.run.smithi110.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:48:52.710 INFO:teuthology.orchestra.run.smithi110.stderr:Error: No packages marked for reinstall. 2023-06-22T17:48:52.757 INFO:teuthology.orchestra.run.smithi116.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-22T17:48:52.757 INFO:teuthology.orchestra.run.smithi116.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:48:52.757 INFO:teuthology.orchestra.run.smithi116.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:48:52.760 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum reinstall -y kernel || true 2023-06-22T17:48:52.769 INFO:teuthology.orchestra.run.smithi116.stderr:Error: No packages marked for reinstall. 2023-06-22T17:48:52.818 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2023-06-22T17:48:52.942 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:52.942 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:52.943 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:52.943 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-22T17:48:52.943 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-22T17:48:52.943 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-06-22T17:48:52.943 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2023-06-22T17:48:52.944 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2023-06-22T17:48:53.067 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:53.068 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:53.068 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2023-06-22T17:48:53.095 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2023-06-22T17:48:53.096 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2023-06-22T17:48:53.096 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.02 2023-06-22T17:48:53.096 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 99.el8_4.1 2023-06-22T17:48:53.096 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2023-06-22T17:48:53.096 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-22T17:48:53.097 INFO:teuthology.orchestra.run.smithi116.stdout:Group : System Environment/Base 2023-06-22T17:48:53.097 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 9264631 2023-06-22T17:48:53.097 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2023-06-22T17:48:53.097 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-22T17:48:53.097 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-22T17:48:53.098 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-22T17:48:53.098 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-22T17:48:53.098 INFO:teuthology.orchestra.run.smithi116.stdout:Relocations : (not relocatable) 2023-06-22T17:48:53.098 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : Red Hat, Inc. 2023-06-22T17:48:53.098 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : Red Hat, Inc. 2023-06-22T17:48:53.098 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-22T17:48:53.099 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2023-06-22T17:48:53.099 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2023-06-22T17:48:53.099 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T17:48:53.099 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-22T17:48:53.099 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-22T17:48:53.099 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-22T17:48:53.100 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2023-06-22T17:48:53.100 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T17:48:53.100 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2023-06-22T17:48:53.101 INFO:teuthology.orchestra.run.smithi110.stdout:Updating Subscription Management repositories. 2023-06-22T17:48:53.101 INFO:teuthology.orchestra.run.smithi110.stdout:Unable to read consumer identity 2023-06-22T17:48:53.101 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-22T17:48:53.102 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:53.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-22T17:48:53.381 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:03 ago on Thu 22 Jun 2023 05:48:50 PM UTC. 2023-06-22T17:48:53.521 INFO:teuthology.orchestra.run.smithi110.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-22T17:48:53.521 INFO:teuthology.orchestra.run.smithi110.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:48:53.522 INFO:teuthology.orchestra.run.smithi110.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-22T17:48:53.540 INFO:teuthology.orchestra.run.smithi110.stderr:Error: No packages marked for reinstall. 2023-06-22T17:48:53.595 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2023-06-22T17:48:53.720 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:53.721 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:53.721 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:53.721 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-22T17:48:53.721 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-22T17:48:53.721 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-06-22T17:48:53.721 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2023-06-22T17:48:53.722 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2023-06-22T17:48:53.846 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:53.847 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:53.847 DEBUG:teuthology.orchestra.run.smithi110:> sudo rpm -qi grub2-tools 2023-06-22T17:48:53.874 INFO:teuthology.orchestra.run.smithi110.stdout:Name : grub2-tools 2023-06-22T17:48:53.875 INFO:teuthology.orchestra.run.smithi110.stdout:Epoch : 1 2023-06-22T17:48:53.875 INFO:teuthology.orchestra.run.smithi110.stdout:Version : 2.02 2023-06-22T17:48:53.875 INFO:teuthology.orchestra.run.smithi110.stdout:Release : 99.el8_4.1 2023-06-22T17:48:53.875 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: x86_64 2023-06-22T17:48:53.875 INFO:teuthology.orchestra.run.smithi110.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-22T17:48:53.876 INFO:teuthology.orchestra.run.smithi110.stdout:Group : System Environment/Base 2023-06-22T17:48:53.876 INFO:teuthology.orchestra.run.smithi110.stdout:Size : 9264631 2023-06-22T17:48:53.876 INFO:teuthology.orchestra.run.smithi110.stdout:License : GPLv3+ 2023-06-22T17:48:53.876 INFO:teuthology.orchestra.run.smithi110.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-22T17:48:53.876 INFO:teuthology.orchestra.run.smithi110.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-22T17:48:53.876 INFO:teuthology.orchestra.run.smithi110.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-22T17:48:53.876 INFO:teuthology.orchestra.run.smithi110.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-22T17:48:53.877 INFO:teuthology.orchestra.run.smithi110.stdout:Relocations : (not relocatable) 2023-06-22T17:48:53.877 INFO:teuthology.orchestra.run.smithi110.stdout:Packager : Red Hat, Inc. 2023-06-22T17:48:53.877 INFO:teuthology.orchestra.run.smithi110.stdout:Vendor : Red Hat, Inc. 2023-06-22T17:48:53.877 INFO:teuthology.orchestra.run.smithi110.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-22T17:48:53.877 INFO:teuthology.orchestra.run.smithi110.stdout:Summary : Support tools for GRUB. 2023-06-22T17:48:53.877 INFO:teuthology.orchestra.run.smithi110.stdout:Description : 2023-06-22T17:48:53.877 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T17:48:53.878 INFO:teuthology.orchestra.run.smithi110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-22T17:48:53.878 INFO:teuthology.orchestra.run.smithi110.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-22T17:48:53.878 INFO:teuthology.orchestra.run.smithi110.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-22T17:48:53.878 INFO:teuthology.orchestra.run.smithi110.stdout:hardware devices. 2023-06-22T17:48:53.878 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T17:48:53.878 INFO:teuthology.orchestra.run.smithi110.stdout:This subpackage provides tools for support of all platforms. 2023-06-22T17:48:53.879 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-22T17:48:53.880 INFO:teuthology.task.kernel:Updating grub on smithi110 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:53.881 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-22T17:48:54.492 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2023-06-22T17:48:55.318 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2023-06-22T17:48:56.603 INFO:teuthology.orchestra.run.smithi116.stderr:done 2023-06-22T17:48:56.603 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-06-22T17:48:56.618 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.1s8xJiFpbk 2023-06-22T17:48:56.618 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1s8xJiFpbk 2023-06-22T17:48:56.700 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.1s8xJiFpbk 2023-06-22T17:48:56.855 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.1s8xJiFpbk is 6KB 2023-06-22T17:48:56.903 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.1s8xJiFpbk 2023-06-22T17:48:56.916 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2023-06-22T17:48:56.980 INFO:teuthology.orchestra.run.smithi116.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-22T17:48:56.980 INFO:teuthology.orchestra.run.smithi116.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-22T17:48:56.980 INFO:teuthology.orchestra.run.smithi116.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-22T17:48:56.981 INFO:teuthology.orchestra.run.smithi116.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-22T17:48:56.982 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:57.174 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2023-06-22T17:48:57.572 INFO:teuthology.orchestra.run.smithi110.stderr:done 2023-06-22T17:48:57.574 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-06-22T17:48:57.588 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.YPtjUxZm7s 2023-06-22T17:48:57.588 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YPtjUxZm7s 2023-06-22T17:48:57.659 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.YPtjUxZm7s 2023-06-22T17:48:57.823 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.YPtjUxZm7s is 6KB 2023-06-22T17:48:57.871 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.YPtjUxZm7s 2023-06-22T17:48:57.885 DEBUG:teuthology.orchestra.run.smithi110:> sudo /bin/ls /boot/loader/entries || true 2023-06-22T17:48:57.956 INFO:teuthology.orchestra.run.smithi110.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-22T17:48:57.956 INFO:teuthology.orchestra.run.smithi110.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-22T17:48:57.956 INFO:teuthology.orchestra.run.smithi110.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-22T17:48:57.957 INFO:teuthology.orchestra.run.smithi110.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-22T17:48:57.958 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-22T17:48:58.142 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2023-06-22T17:49:27.207 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-06-22T17:49:27.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-06-22T17:49:27.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:49:28.146 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-06-22T17:49:28.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-06-22T17:49:28.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:49:45.712 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2023-06-22T17:49:46.480 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-06-22T17:50:15.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-06-22T17:50:15.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:50:16.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-06-22T17:50:16.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:50:33.069 DEBUG:teuthology.orchestra.run.smithi110:> true 2023-06-22T17:50:33.321 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-06-22T17:50:33.322 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-22T17:50:33.322 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2023-06-22T17:50:33.376 INFO:teuthology.orchestra.run.smithi110.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-22T17:50:33.377 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:50:33.377 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-22T17:50:33.377 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-22T17:50:34.096 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2023-06-22T17:50:34.378 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-06-22T17:50:34.379 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-22T17:50:34.444 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2023-06-22T17:50:34.461 DEBUG:teuthology.parallel:result is None 2023-06-22T17:51:04.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-06-22T17:51:04.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T17:51:04.505 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-06-22T17:51:04.782 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-06-22T17:51:04.782 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-22T17:51:04.783 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2023-06-22T17:51:04.797 INFO:teuthology.orchestra.run.smithi116.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-22T17:51:04.798 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-22T17:51:04.798 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-22T17:51:04.799 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-22T17:51:05.800 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-06-22T17:51:05.801 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-22T17:51:05.875 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2023-06-22T17:51:05.895 DEBUG:teuthology.parallel:result is None 2023-06-22T17:51:05.895 INFO:teuthology.run_tasks:Running task internal.base... 2023-06-22T17:51:05.902 INFO:teuthology.task.internal:Creating test directory... 2023-06-22T17:51:05.902 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-22T17:51:05.906 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-22T17:51:05.923 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-06-22T17:51:05.938 INFO:teuthology.run_tasks:Running task internal.archive... 2023-06-22T17:51:05.971 INFO:teuthology.task.internal:Creating archive directory... 2023-06-22T17:51:05.971 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-22T17:51:05.974 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-22T17:51:06.006 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-06-22T17:51:06.017 INFO:teuthology.task.internal:Enabling coredump saving... 2023-06-22T17:51:06.018 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-22T17:51:06.044 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-22T17:51:06.082 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T17:51:06.093 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T17:51:06.093 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T17:51:06.106 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-22T17:51:06.107 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-06-22T17:51:06.114 INFO:teuthology.task.internal:Configuring sudo... 2023-06-22T17:51:06.114 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-22T17:51:06.116 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-22T17:51:06.159 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-06-22T17:51:06.168 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-06-22T17:51:06.168 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-22T17:51:06.184 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-22T17:51:06.214 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T17:51:06.283 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T17:51:06.322 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T17:51:06.407 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T17:51:06.444 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T17:51:06.444 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-22T17:51:06.508 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T17:51:06.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-22T17:51:06.590 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T17:51:06.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-22T17:51:06.711 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T17:51:06.711 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-22T17:51:06.774 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2023-06-22T17:51:06.776 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2023-06-22T17:51:06.845 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T17:51:06.868 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T17:51:07.308 INFO:teuthology.run_tasks:Running task internal.timer... 2023-06-22T17:51:07.316 INFO:teuthology.task.internal:Starting timer... 2023-06-22T17:51:07.316 INFO:teuthology.run_tasks:Running task pcp... 2023-06-22T17:51:07.391 INFO:teuthology.run_tasks:Running task selinux... 2023-06-22T17:51:07.457 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-06-22T17:51:07.458 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2023-06-22T17:51:07.516 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: [ OK ] 2023-06-22T17:51:07.517 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2023-06-22T17:51:07.576 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: [ OK ] 2023-06-22T17:51:07.577 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-06-22T17:51:07.577 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2023-06-22T17:51:07.604 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2023-06-22T17:51:07.604 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2023-06-22T17:51:07.623 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2023-06-22T17:51:07.623 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi110.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi116.front.sepia.ceph.com': 'permissive'} 2023-06-22T17:51:07.624 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T17:51:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:51:07.670 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T17:51:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T17:51:07.697 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-06-22T17:51:07.697 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2023-06-22T17:51:07.737 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2023-06-22T17:51:07.763 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-06-22T17:51:07.779 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-06-22T17:51:07.779 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-06-22T17:51:07.807 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-06-22T17:51:07.809 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi110.front.sepia.ceph.com,smithi116.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-06-22T18:03:16.842 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi116.front.sepia.ceph.com')] 2023-06-22T18:03:16.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-06-22T18:03:16.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T18:03:16.916 DEBUG:teuthology.orchestra.run.smithi110:> true 2023-06-22T18:03:16.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2023-06-22T18:03:16.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T18:03:17.052 DEBUG:teuthology.orchestra.run.smithi116:> true 2023-06-22T18:03:17.118 INFO:teuthology.run_tasks:Running task clock... 2023-06-22T18:03:17.134 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-06-22T18:03:17.134 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-22T18:03:17.135 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T18:03:17.137 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-22T18:03:17.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T18:03:17.163 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-22T18:03:17.177 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-22T18:03:17.186 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-22T18:03:17.201 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-22T18:03:17.206 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2023-06-22T18:03:17.216 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2023-06-22T18:03:17.230 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-22T18:03:17.233 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2023-06-22T18:03:17.243 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-22T18:03:17.244 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2023-06-22T18:03:17.258 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-22T18:03:17.272 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-22T18:03:17.305 INFO:teuthology.orchestra.run.smithi110.stderr:bash: ntpq: command not found 2023-06-22T18:03:17.307 INFO:teuthology.orchestra.run.smithi110.stdout:210 Number of sources = 4 2023-06-22T18:03:17.307 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T18:03:17.308 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2023-06-22T18:03:17.308 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T18:03:17.308 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T18:03:17.308 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T18:03:17.308 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T18:03:17.339 INFO:teuthology.orchestra.run.smithi116.stderr:bash: ntpq: command not found 2023-06-22T18:03:17.341 INFO:teuthology.orchestra.run.smithi116.stdout:210 Number of sources = 4 2023-06-22T18:03:17.341 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T18:03:17.341 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2023-06-22T18:03:17.342 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T18:03:17.342 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T18:03:17.342 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T18:03:17.343 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T18:03:17.343 INFO:teuthology.run_tasks:Running task install... 2023-06-22T18:03:17.355 DEBUG:teuthology.task.install:project ceph 2023-06-22T18:03:17.356 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}} 2023-06-22T18:03:17.356 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'} 2023-06-22T18:03:17.356 INFO:teuthology.task.install:Using flavor: default 2023-06-22T18:03:17.362 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-06-22T18:03:17.362 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-06-22T18:03:17.363 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-22T18:03:17.363 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T18:03:17.365 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-22T18:03:17.366 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T18:03:18.803 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-22T18:03:18.804 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-22T18:03:20.919 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-22T18:03:20.920 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:03:20.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-22T18:03:20.946 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-22T18:03:20.946 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-22T18:03:21.011 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-22T18:03:21.086 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-22T18:03:21.151 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2023-06-22T18:03:21.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2023-06-22T18:03:21.548 INFO:teuthology.orchestra.run.smithi116.stdout:Updating Subscription Management repositories. 2023-06-22T18:03:21.829 INFO:teuthology.orchestra.run.smithi116.stdout:44 files removed 2023-06-22T18:03:21.880 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-22T18:03:22.236 INFO:teuthology.orchestra.run.smithi116.stdout:Updating Subscription Management repositories. 2023-06-22T18:03:27.274 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-22T18:03:27.275 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-22T18:03:29.084 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-22T18:03:29.084 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:03:29.085 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-22T18:03:29.116 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-22T18:03:29.116 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-22T18:03:29.185 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-22T18:03:29.263 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-22T18:03:29.329 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2023-06-22T18:03:29.330 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2023-06-22T18:03:29.721 INFO:teuthology.orchestra.run.smithi110.stdout:Updating Subscription Management repositories. 2023-06-22T18:03:29.995 INFO:teuthology.orchestra.run.smithi110.stdout:44 files removed 2023-06-22T18:03:30.044 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-22T18:03:30.393 INFO:teuthology.orchestra.run.smithi110.stdout:Updating Subscription Management repositories. 2023-06-22T18:04:03.201 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 2.4 kB/s | 78 kB 00:32 2023-06-22T18:04:20.685 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 538 B/s | 9.2 kB 00:17 2023-06-22T18:04:20.971 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 5.7 kB/s | 1.4 kB 00:00 2023-06-22T18:04:23.223 INFO:teuthology.orchestra.run.smithi110.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2023-06-22T18:04:24.419 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-06-22T18:04:26.885 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 1.2 kB/s | 78 kB 01:04 2023-06-22T18:04:27.233 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 30 kB/s | 9.2 kB 00:00 2023-06-22T18:04:27.570 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 4.7 kB/s | 1.4 kB 00:00 2023-06-22T18:04:27.760 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 1.7 MB/s | 24 kB 00:00 2023-06-22T18:04:28.173 INFO:teuthology.orchestra.run.smithi110.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 115 MB/s | 38 MB 00:00 2023-06-22T18:04:30.015 INFO:teuthology.orchestra.run.smithi116.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-22T18:04:30.907 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2023-06-22T18:04:33.878 INFO:teuthology.orchestra.run.smithi110.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 MB/s | 33 MB 00:00 2023-06-22T18:04:34.099 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 4.8 MB/s | 24 kB 00:00 2023-06-22T18:04:34.492 INFO:teuthology.orchestra.run.smithi116.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 123 MB/s | 38 MB 00:00 2023-06-22T18:04:39.977 INFO:teuthology.orchestra.run.smithi116.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 120 MB/s | 33 MB 00:00 2023-06-22T18:04:41.960 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T18:04:41.961 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T18:04:41.972 INFO:teuthology.orchestra.run.smithi110.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-22T18:04:41.973 INFO:teuthology.orchestra.run.smithi110.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-22T18:04:41.974 INFO:teuthology.orchestra.run.smithi110.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-22T18:04:41.975 INFO:teuthology.orchestra.run.smithi110.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-22T18:04:41.976 INFO:teuthology.orchestra.run.smithi110.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-22T18:04:41.979 INFO:teuthology.orchestra.run.smithi110.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-22T18:04:41.979 INFO:teuthology.orchestra.run.smithi110.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-22T18:04:42.482 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-06-22T18:04:42.494 INFO:teuthology.orchestra.run.smithi110.stdout:====================================================================================================================== 2023-06-22T18:04:42.494 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2023-06-22T18:04:42.494 INFO:teuthology.orchestra.run.smithi110.stdout:====================================================================================================================== 2023-06-22T18:04:42.494 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-06-22T18:04:42.494 INFO:teuthology.orchestra.run.smithi110.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-22T18:04:42.495 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-22T18:04:42.495 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-22T18:04:42.495 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-22T18:04:42.495 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-22T18:04:42.495 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-22T18:04:42.495 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-22T18:04:42.495 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-22T18:04:42.496 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-22T18:04:42.496 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-22T18:04:42.496 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-22T18:04:42.496 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-22T18:04:42.496 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-22T18:04:42.496 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-22T18:04:42.496 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-22T18:04:42.496 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-22T18:04:42.497 INFO:teuthology.orchestra.run.smithi110.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-22T18:04:42.497 INFO:teuthology.orchestra.run.smithi110.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-22T18:04:42.497 INFO:teuthology.orchestra.run.smithi110.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-22T18:04:42.497 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-22T18:04:42.497 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-22T18:04:42.497 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-22T18:04:42.497 INFO:teuthology.orchestra.run.smithi110.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-22T18:04:42.498 INFO:teuthology.orchestra.run.smithi110.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-22T18:04:42.498 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-22T18:04:42.498 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-22T18:04:42.498 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-22T18:04:42.498 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-22T18:04:42.498 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-22T18:04:42.498 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-22T18:04:42.498 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-22T18:04:42.499 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-22T18:04:42.499 INFO:teuthology.orchestra.run.smithi110.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-22T18:04:42.499 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2023-06-22T18:04:42.499 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-22T18:04:42.499 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-22T18:04:42.499 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2023-06-22T18:04:42.499 INFO:teuthology.orchestra.run.smithi110.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-22T18:04:42.500 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-22T18:04:42.500 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-22T18:04:42.500 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-22T18:04:42.500 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-22T18:04:42.500 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-22T18:04:42.500 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-22T18:04:42.500 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-22T18:04:42.500 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-22T18:04:42.501 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-22T18:04:42.501 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-22T18:04:42.501 INFO:teuthology.orchestra.run.smithi110.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-22T18:04:42.501 INFO:teuthology.orchestra.run.smithi110.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-22T18:04:42.501 INFO:teuthology.orchestra.run.smithi110.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-22T18:04:42.501 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-22T18:04:42.501 INFO:teuthology.orchestra.run.smithi110.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-22T18:04:42.502 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-22T18:04:42.502 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-22T18:04:42.502 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-22T18:04:42.502 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-22T18:04:42.502 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-22T18:04:42.502 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-22T18:04:42.502 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-22T18:04:42.502 INFO:teuthology.orchestra.run.smithi110.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-22T18:04:42.503 INFO:teuthology.orchestra.run.smithi110.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-22T18:04:42.503 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-22T18:04:42.503 INFO:teuthology.orchestra.run.smithi110.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-22T18:04:42.503 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-22T18:04:42.503 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-22T18:04:42.503 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-22T18:04:42.503 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-22T18:04:42.503 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-22T18:04:42.504 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-22T18:04:42.504 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-22T18:04:42.504 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-22T18:04:42.504 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-22T18:04:42.504 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-22T18:04:42.504 INFO:teuthology.orchestra.run.smithi110.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-22T18:04:42.505 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-22T18:04:42.505 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-22T18:04:42.505 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-22T18:04:42.505 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-22T18:04:42.505 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-22T18:04:42.505 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-22T18:04:42.505 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-22T18:04:42.505 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-22T18:04:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-22T18:04:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-22T18:04:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-22T18:04:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-22T18:04:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T18:04:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-22T18:04:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-22T18:04:42.506 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-22T18:04:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-22T18:04:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-22T18:04:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-22T18:04:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-22T18:04:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-22T18:04:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-22T18:04:42.507 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-22T18:04:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-22T18:04:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-22T18:04:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-22T18:04:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-22T18:04:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-22T18:04:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-22T18:04:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T18:04:42.508 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-22T18:04:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-22T18:04:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-22T18:04:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-22T18:04:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-22T18:04:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-22T18:04:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-22T18:04:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-22T18:04:42.509 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-22T18:04:42.510 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-22T18:04:42.510 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-22T18:04:42.510 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-22T18:04:42.510 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-22T18:04:42.510 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-22T18:04:42.510 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-22T18:04:42.510 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-22T18:04:42.510 INFO:teuthology.orchestra.run.smithi110.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-22T18:04:42.511 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:04:42.511 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-06-22T18:04:42.511 INFO:teuthology.orchestra.run.smithi110.stdout:====================================================================================================================== 2023-06-22T18:04:42.511 INFO:teuthology.orchestra.run.smithi110.stdout:Install 118 Packages 2023-06-22T18:04:42.511 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 2 Packages 2023-06-22T18:04:42.511 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:04:42.512 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 201 M 2023-06-22T18:04:42.512 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-06-22T18:04:43.103 INFO:teuthology.orchestra.run.smithi110.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 39 kB/s | 6.4 kB 00:00 2023-06-22T18:04:43.365 INFO:teuthology.orchestra.run.smithi110.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 3.3 MB/s | 873 kB 00:00 2023-06-22T18:04:43.403 INFO:teuthology.orchestra.run.smithi110.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 4.2 MB/s | 162 kB 00:00 2023-06-22T18:04:43.694 INFO:teuthology.orchestra.run.smithi110.stdout:(4/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 7.9 MB/s | 2.3 MB 00:00 2023-06-22T18:04:43.760 INFO:teuthology.orchestra.run.smithi110.stdout:(5/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 6.3 MB/s | 5.2 MB 00:00 2023-06-22T18:04:43.892 INFO:teuthology.orchestra.run.smithi110.stdout:(6/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 7.9 MB/s | 1.6 MB 00:00 2023-06-22T18:04:44.182 INFO:teuthology.orchestra.run.smithi110.stdout:(7/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 9.3 MB/s | 3.9 MB 00:00 2023-06-22T18:04:45.560 INFO:teuthology.orchestra.run.smithi110.stdout:(8/120): ceph-common-17.2.6-719.g42ddc46e.el8.x 8.9 MB/s | 23 MB 00:02 2023-06-22T18:04:45.608 INFO:teuthology.orchestra.run.smithi110.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 510 kB/s | 24 kB 00:00 2023-06-22T18:04:45.908 INFO:teuthology.orchestra.run.smithi110.stdout:(10/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 9.7 MB/s | 17 MB 00:01 2023-06-22T18:04:46.035 INFO:teuthology.orchestra.run.smithi110.stdout:(11/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 7.8 MB/s | 17 MB 00:02 2023-06-22T18:04:46.037 INFO:teuthology.orchestra.run.smithi110.stdout:(12/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 1.9 MB/s | 255 kB 00:00 2023-06-22T18:04:46.091 INFO:teuthology.orchestra.run.smithi110.stdout:(13/120): libcephfs-devel-17.2.6-719.g42ddc46e. 444 kB/s | 25 kB 00:00 2023-06-22T18:04:46.120 INFO:teuthology.orchestra.run.smithi110.stdout:(14/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 9.4 MB/s | 794 kB 00:00 2023-06-22T18:04:46.153 INFO:teuthology.orchestra.run.smithi110.stdout:(15/120): libcephsqlite-17.2.6-719.g42ddc46e.el 2.8 MB/s | 174 kB 00:00 2023-06-22T18:04:46.160 INFO:teuthology.orchestra.run.smithi110.stdout:(16/120): librados-devel-17.2.6-719.g42ddc46e.e 3.2 MB/s | 130 kB 00:00 2023-06-22T18:04:46.261 INFO:teuthology.orchestra.run.smithi110.stdout:(17/120): libradosstriper1-17.2.6-719.g42ddc46e 4.6 MB/s | 506 kB 00:00 2023-06-22T18:04:46.320 INFO:teuthology.orchestra.run.smithi110.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 756 kB/s | 45 kB 00:00 2023-06-22T18:04:46.382 INFO:teuthology.orchestra.run.smithi110.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 1.8 MB/s | 115 kB 00:00 2023-06-22T18:04:46.448 INFO:teuthology.orchestra.run.smithi110.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 2.8 MB/s | 190 kB 00:00 2023-06-22T18:04:46.521 INFO:teuthology.orchestra.run.smithi110.stdout:(21/120): python3-rados-17.2.6-719.g42ddc46e.el 5.0 MB/s | 371 kB 00:00 2023-06-22T18:04:46.593 INFO:teuthology.orchestra.run.smithi110.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 4.8 MB/s | 357 kB 00:00 2023-06-22T18:04:46.625 INFO:teuthology.orchestra.run.smithi110.stdout:(23/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 12 MB/s | 5.5 MB 00:00 2023-06-22T18:04:46.644 INFO:teuthology.orchestra.run.smithi110.stdout:(24/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 2.1 MB/s | 107 kB 00:00 2023-06-22T18:04:46.669 INFO:teuthology.orchestra.run.smithi110.stdout:(25/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 2.0 MB/s | 91 kB 00:00 2023-06-22T18:04:46.726 INFO:teuthology.orchestra.run.smithi110.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 3.0 MB/s | 173 kB 00:00 2023-06-22T18:04:46.780 INFO:teuthology.orchestra.run.smithi110.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 431 kB/s | 23 kB 00:00 2023-06-22T18:04:46.839 INFO:teuthology.orchestra.run.smithi110.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 2.0 MB/s | 121 kB 00:00 2023-06-22T18:04:47.012 INFO:teuthology.orchestra.run.smithi110.stdout:(29/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 9.8 MB/s | 1.7 MB 00:00 2023-06-22T18:04:47.035 INFO:teuthology.orchestra.run.smithi110.stdout:(30/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 8.9 MB/s | 3.5 MB 00:00 2023-06-22T18:04:47.107 INFO:teuthology.orchestra.run.smithi110.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 3.2 MB/s | 238 kB 00:00 2023-06-22T18:04:47.176 INFO:teuthology.orchestra.run.smithi110.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 694 kB/s | 47 kB 00:00 2023-06-22T18:04:47.236 INFO:teuthology.orchestra.run.smithi110.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 239 kB/s | 14 kB 00:00 2023-06-22T18:04:47.306 INFO:teuthology.orchestra.run.smithi110.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 3.7 MB/s | 264 kB 00:00 2023-06-22T18:04:47.370 INFO:teuthology.orchestra.run.smithi110.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.2 MB/s | 80 kB 00:00 2023-06-22T18:04:47.434 INFO:teuthology.orchestra.run.smithi110.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 281 kB/s | 18 kB 00:00 2023-06-22T18:04:47.569 INFO:teuthology.orchestra.run.smithi110.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 2.2 MB/s | 306 kB 00:00 2023-06-22T18:04:47.603 INFO:teuthology.orchestra.run.smithi110.stdout:(38/120): ceph-mgr-diskprediction-local-17.2.6- 13 MB/s | 7.4 MB 00:00 2023-06-22T18:04:47.604 INFO:teuthology.orchestra.run.smithi110.stdout:(39/120): liboath-2.6.2-3.el8.x86_64.rpm 1.6 MB/s | 59 kB 00:00 2023-06-22T18:04:47.671 INFO:teuthology.orchestra.run.smithi110.stdout:(40/120): python3-asyncssh-2.7.0-2.el8.noarch.r 7.0 MB/s | 476 kB 00:00 2023-06-22T18:04:47.686 INFO:teuthology.orchestra.run.smithi110.stdout:(41/120): libunwind-1.3.1-3.el8.x86_64.rpm 909 kB/s | 75 kB 00:00 2023-06-22T18:04:47.692 INFO:teuthology.orchestra.run.smithi110.stdout:(42/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 2.0 MB/s | 44 kB 00:00 2023-06-22T18:04:47.715 INFO:teuthology.orchestra.run.smithi110.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 1.4 MB/s | 33 kB 00:00 2023-06-22T18:04:47.737 INFO:teuthology.orchestra.run.smithi110.stdout:(44/120): python3-certifi-2018.10.15-7.el8.noar 763 kB/s | 16 kB 00:00 2023-06-22T18:04:47.737 INFO:teuthology.orchestra.run.smithi110.stdout:(45/120): python3-beautifulsoup4-4.6.3-2.el8.1. 3.5 MB/s | 185 kB 00:00 2023-06-22T18:04:47.760 INFO:teuthology.orchestra.run.smithi110.stdout:(46/120): python3-cheroot-8.5.2-1.el8.noarch.rp 7.3 MB/s | 173 kB 00:00 2023-06-22T18:04:47.783 INFO:teuthology.orchestra.run.smithi110.stdout:(47/120): python3-google-auth-1.1.1-10.el8.noar 3.6 MB/s | 81 kB 00:00 2023-06-22T18:04:47.805 INFO:teuthology.orchestra.run.smithi110.stdout:(48/120): python3-jaraco-6.2-6.el8.noarch.rpm 506 kB/s | 11 kB 00:00 2023-06-22T18:04:47.818 INFO:teuthology.orchestra.run.smithi110.stdout:(49/120): python3-cherrypy-18.4.0-1.el8.noarch. 4.7 MB/s | 384 kB 00:00 2023-06-22T18:04:47.826 INFO:teuthology.orchestra.run.smithi110.stdout:(50/120): python3-jaraco-functools-2.0-4.el8.no 832 kB/s | 18 kB 00:00 2023-06-22T18:04:47.826 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T18:04:47.827 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-22T18:04:47.838 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-22T18:04:47.838 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-22T18:04:47.839 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-22T18:04:47.840 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-22T18:04:47.841 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-22T18:04:47.844 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-22T18:04:47.844 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-22T18:04:47.847 INFO:teuthology.orchestra.run.smithi110.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 2.3 MB/s | 49 kB 00:00 2023-06-22T18:04:47.869 INFO:teuthology.orchestra.run.smithi110.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 2.7 MB/s | 59 kB 00:00 2023-06-22T18:04:47.890 INFO:teuthology.orchestra.run.smithi110.stdout:(53/120): python3-natsort-7.1.1-2.el8.noarch.rp 2.8 MB/s | 60 kB 00:00 2023-06-22T18:04:47.937 INFO:teuthology.orchestra.run.smithi110.stdout:(54/120): python3-pecan-1.3.2-9.el8.noarch.rpm 6.0 MB/s | 283 kB 00:00 2023-06-22T18:04:47.969 INFO:teuthology.orchestra.run.smithi110.stdout:(55/120): python3-portend-2.6-1.el8.noarch.rpm 481 kB/s | 16 kB 00:00 2023-06-22T18:04:48.002 INFO:teuthology.orchestra.run.smithi110.stdout:(56/120): python3-repoze-lru-0.7-6.el8.noarch.r 1.0 MB/s | 34 kB 00:00 2023-06-22T18:04:48.022 INFO:teuthology.orchestra.run.smithi110.stdout:(57/120): python3-kubernetes-11.0.0-6.el8.noarc 5.7 MB/s | 1.2 MB 00:00 2023-06-22T18:04:48.041 INFO:teuthology.orchestra.run.smithi110.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 5.0 MB/s | 196 kB 00:00 2023-06-22T18:04:48.057 INFO:teuthology.orchestra.run.smithi110.stdout:(59/120): python3-rsa-4.8-1.el8.noarch.rpm 1.7 MB/s | 61 kB 00:00 2023-06-22T18:04:48.082 INFO:teuthology.orchestra.run.smithi110.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 462 kB/s | 19 kB 00:00 2023-06-22T18:04:48.093 INFO:teuthology.orchestra.run.smithi110.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 664 kB/s | 24 kB 00:00 2023-06-22T18:04:48.119 INFO:teuthology.orchestra.run.smithi110.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 807 kB/s | 29 kB 00:00 2023-06-22T18:04:48.127 INFO:teuthology.orchestra.run.smithi110.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 811 kB/s | 27 kB 00:00 2023-06-22T18:04:48.197 INFO:teuthology.orchestra.run.smithi110.stdout:(64/120): python3-waitress-1.4.3-1.el8.noarch.r 3.2 MB/s | 255 kB 00:00 2023-06-22T18:04:48.207 INFO:teuthology.orchestra.run.smithi110.stdout:(65/120): python3-webob-1.8.5-1.el8.1.noarch.rp 3.1 MB/s | 251 kB 00:00 2023-06-22T18:04:48.237 INFO:teuthology.orchestra.run.smithi110.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 1.5 MB/s | 61 kB 00:00 2023-06-22T18:04:48.248 INFO:teuthology.orchestra.run.smithi110.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 2.1 MB/s | 85 kB 00:00 2023-06-22T18:04:48.289 INFO:teuthology.orchestra.run.smithi110.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 444 kB/s | 23 kB 00:00 2023-06-22T18:04:48.293 INFO:teuthology.orchestra.run.smithi110.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 19 MB/s | 70 kB 00:00 2023-06-22T18:04:48.327 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2023-06-22T18:04:48.337 INFO:teuthology.orchestra.run.smithi110.stdout:(70/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 429 kB/s | 19 kB 00:00 2023-06-22T18:04:48.338 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================================================== 2023-06-22T18:04:48.338 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2023-06-22T18:04:48.338 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================================================== 2023-06-22T18:04:48.339 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2023-06-22T18:04:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-22T18:04:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-22T18:04:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-22T18:04:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-22T18:04:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-22T18:04:48.339 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-22T18:04:48.340 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-22T18:04:48.340 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-22T18:04:48.340 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-22T18:04:48.340 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-22T18:04:48.340 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-22T18:04:48.340 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-22T18:04:48.340 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-22T18:04:48.341 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-22T18:04:48.341 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-22T18:04:48.341 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-22T18:04:48.341 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-22T18:04:48.341 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-22T18:04:48.341 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-22T18:04:48.341 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-22T18:04:48.341 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-22T18:04:48.342 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-22T18:04:48.342 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-22T18:04:48.342 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-22T18:04:48.342 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-22T18:04:48.342 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-22T18:04:48.342 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-22T18:04:48.342 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-22T18:04:48.343 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-22T18:04:48.343 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-22T18:04:48.343 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-22T18:04:48.343 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-22T18:04:48.343 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-22T18:04:48.343 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2023-06-22T18:04:48.343 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-22T18:04:48.343 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-22T18:04:48.344 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2023-06-22T18:04:48.344 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-22T18:04:48.344 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-22T18:04:48.344 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-22T18:04:48.344 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-22T18:04:48.344 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-22T18:04:48.344 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-22T18:04:48.345 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-22T18:04:48.345 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-22T18:04:48.345 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-22T18:04:48.345 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-22T18:04:48.345 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-22T18:04:48.345 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-22T18:04:48.345 INFO:teuthology.orchestra.run.smithi116.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-22T18:04:48.346 INFO:teuthology.orchestra.run.smithi116.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-22T18:04:48.346 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-22T18:04:48.346 INFO:teuthology.orchestra.run.smithi116.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-22T18:04:48.346 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-22T18:04:48.346 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-22T18:04:48.346 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-22T18:04:48.346 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-22T18:04:48.346 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-22T18:04:48.347 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-22T18:04:48.347 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-22T18:04:48.347 INFO:teuthology.orchestra.run.smithi116.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-22T18:04:48.347 INFO:teuthology.orchestra.run.smithi116.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-22T18:04:48.347 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-22T18:04:48.347 INFO:teuthology.orchestra.run.smithi116.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-22T18:04:48.347 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-22T18:04:48.347 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-22T18:04:48.348 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-22T18:04:48.348 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-22T18:04:48.348 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-22T18:04:48.348 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-22T18:04:48.348 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-22T18:04:48.348 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-22T18:04:48.348 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-22T18:04:48.349 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-22T18:04:48.349 INFO:teuthology.orchestra.run.smithi116.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-22T18:04:48.349 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-22T18:04:48.349 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-22T18:04:48.349 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-22T18:04:48.349 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-22T18:04:48.349 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-22T18:04:48.350 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-22T18:04:48.350 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-22T18:04:48.350 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-22T18:04:48.350 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-22T18:04:48.350 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-22T18:04:48.350 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-22T18:04:48.350 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-22T18:04:48.350 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T18:04:48.351 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-22T18:04:48.351 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-22T18:04:48.351 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-22T18:04:48.351 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-22T18:04:48.351 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-22T18:04:48.351 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-22T18:04:48.351 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-22T18:04:48.351 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-22T18:04:48.352 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-22T18:04:48.352 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-22T18:04:48.352 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-22T18:04:48.352 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-22T18:04:48.352 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-22T18:04:48.352 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-22T18:04:48.352 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-22T18:04:48.352 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-22T18:04:48.353 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-22T18:04:48.353 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-22T18:04:48.353 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-22T18:04:48.353 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-22T18:04:48.353 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-22T18:04:48.353 INFO:teuthology.orchestra.run.smithi116.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-22T18:04:48.353 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-22T18:04:48.353 INFO:teuthology.orchestra.run.smithi116.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-22T18:04:48.354 INFO:teuthology.orchestra.run.smithi116.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-22T18:04:48.354 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-22T18:04:48.354 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-22T18:04:48.354 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-22T18:04:48.354 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-22T18:04:48.354 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-22T18:04:48.354 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-22T18:04:48.355 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-22T18:04:48.355 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-22T18:04:48.355 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-22T18:04:48.355 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:04:48.355 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2023-06-22T18:04:48.355 INFO:teuthology.orchestra.run.smithi116.stdout:====================================================================================================================== 2023-06-22T18:04:48.356 INFO:teuthology.orchestra.run.smithi116.stdout:Install 118 Packages 2023-06-22T18:04:48.356 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 2 Packages 2023-06-22T18:04:48.356 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:04:48.356 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 201 M 2023-06-22T18:04:48.357 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2023-06-22T18:04:48.363 INFO:teuthology.orchestra.run.smithi110.stdout:(71/120): python3-jwt-1.6.1-2.el8.noarch.rpm 1.6 MB/s | 43 kB 00:00 2023-06-22T18:04:48.389 INFO:teuthology.orchestra.run.smithi110.stdout:(72/120): python3-oauthlib-2.1.0-1.el8.noarch.r 6.0 MB/s | 155 kB 00:00 2023-06-22T18:04:48.415 INFO:teuthology.orchestra.run.smithi110.stdout:(73/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 8.5 MB/s | 220 kB 00:00 2023-06-22T18:04:48.438 INFO:teuthology.orchestra.run.smithi110.stdout:(74/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 2.0 MB/s | 48 kB 00:00 2023-06-22T18:04:48.463 INFO:teuthology.orchestra.run.smithi110.stdout:(75/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 9.0 MB/s | 229 kB 00:00 2023-06-22T18:04:48.486 INFO:teuthology.orchestra.run.smithi110.stdout:(76/120): python3-requests-oauthlib-1.0.0-1.el8 1.9 MB/s | 43 kB 00:00 2023-06-22T18:04:48.533 INFO:teuthology.orchestra.run.smithi110.stdout:(77/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 49 MB/s | 2.3 MB 00:00 2023-06-22T18:04:48.538 INFO:teuthology.orchestra.run.smithi110.stdout:(78/120): thrift-0.13.0-2.el8.x86_64.rpm 6.0 MB/s | 1.7 MB 00:00 2023-06-22T18:04:48.556 INFO:teuthology.orchestra.run.smithi110.stdout:(79/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 791 kB/s | 18 kB 00:00 2023-06-22T18:04:48.580 INFO:teuthology.orchestra.run.smithi110.stdout:(80/120): fuse-2.9.7-12.el8.x86_64.rpm 3.5 MB/s | 83 kB 00:00 2023-06-22T18:04:48.582 INFO:teuthology.orchestra.run.smithi110.stdout:(81/120): pcre2-devel-10.32-2.el8.x86_64.rpm 13 MB/s | 605 kB 00:00 2023-06-22T18:04:48.620 INFO:teuthology.orchestra.run.smithi110.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 1.4 MB/s | 59 kB 00:00 2023-06-22T18:04:48.622 INFO:teuthology.orchestra.run.smithi110.stdout:(83/120): libquadmath-8.4.1-1.el8.x86_64.rpm 4.2 MB/s | 169 kB 00:00 2023-06-22T18:04:48.634 INFO:teuthology.orchestra.run.smithi116.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 40 kB/s | 6.4 kB 00:00 2023-06-22T18:04:48.642 INFO:teuthology.orchestra.run.smithi110.stdout:(84/120): libcom_err-devel-1.45.6-1.el8.x86_64. 1.7 MB/s | 38 kB 00:00 2023-06-22T18:04:48.646 INFO:teuthology.orchestra.run.smithi110.stdout:(85/120): libsepol-devel-2.9-2.el8.x86_64.rpm 3.6 MB/s | 86 kB 00:00 2023-06-22T18:04:48.895 INFO:teuthology.orchestra.run.smithi116.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 3.3 MB/s | 873 kB 00:00 2023-06-22T18:04:48.936 INFO:teuthology.orchestra.run.smithi116.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 3.9 MB/s | 162 kB 00:00 2023-06-22T18:04:49.083 INFO:teuthology.orchestra.run.smithi110.stdout:(86/120): ceph-test-17.2.6-719.g42ddc46e.el8.x8 15 MB/s | 52 MB 00:03 2023-06-22T18:04:49.085 INFO:teuthology.orchestra.run.smithi110.stdout:(87/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 107 kB/s | 47 kB 00:00 2023-06-22T18:04:49.086 INFO:teuthology.orchestra.run.smithi110.stdout:(88/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 130 kB/s | 58 kB 00:00 2023-06-22T18:04:49.123 INFO:teuthology.orchestra.run.smithi110.stdout:(89/120): libgfortran-8.4.1-1.el8.x86_64.rpm 17 MB/s | 639 kB 00:00 2023-06-22T18:04:49.127 INFO:teuthology.orchestra.run.smithi110.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 1.7 MB/s | 70 kB 00:00 2023-06-22T18:04:49.143 INFO:teuthology.orchestra.run.smithi110.stdout:(91/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 9.2 MB/s | 559 kB 00:00 2023-06-22T18:04:49.170 INFO:teuthology.orchestra.run.smithi110.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 2.2 MB/s | 97 kB 00:00 2023-06-22T18:04:49.171 INFO:teuthology.orchestra.run.smithi110.stdout:(93/120): libselinux-devel-2.9-5.el8.x86_64.rpm 4.1 MB/s | 200 kB 00:00 2023-06-22T18:04:49.183 INFO:teuthology.orchestra.run.smithi110.stdout:(94/120): python3-prettytable-0.7.2-14.el8.noar 3.4 MB/s | 44 kB 00:00 2023-06-22T18:04:49.184 INFO:teuthology.orchestra.run.smithi110.stdout:(95/120): python3-jsonpatch-1.21-2.el8.noarch.r 2.0 MB/s | 27 kB 00:00 2023-06-22T18:04:49.197 INFO:teuthology.orchestra.run.smithi116.stdout:(4/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 8.8 MB/s | 2.3 MB 00:00 2023-06-22T18:04:49.198 INFO:teuthology.orchestra.run.smithi110.stdout:(96/120): jq-1.5-12.el8.x86_64.rpm 11 MB/s | 161 kB 00:00 2023-06-22T18:04:49.202 INFO:teuthology.orchestra.run.smithi110.stdout:(97/120): flex-2.6.1-9.el8.x86_64.rpm 18 MB/s | 320 kB 00:00 2023-06-22T18:04:49.218 INFO:teuthology.orchestra.run.smithi110.stdout:(98/120): python3-werkzeug-0.12.2-4.el8.noarch. 22 MB/s | 457 kB 00:00 2023-06-22T18:04:49.232 INFO:teuthology.orchestra.run.smithi110.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 2.9 MB/s | 39 kB 00:00 2023-06-22T18:04:49.256 INFO:teuthology.orchestra.run.smithi110.stdout:(100/120): bison-3.0.4-10.el8.x86_64.rpm 29 MB/s | 688 kB 00:00 2023-06-22T18:04:49.271 INFO:teuthology.orchestra.run.smithi110.stdout:(101/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 16 MB/s | 259 kB 00:00 2023-06-22T18:04:49.292 INFO:teuthology.orchestra.run.smithi110.stdout:(102/120): python3-babel-2.5.1-5.el8.noarch.rpm 54 MB/s | 4.8 MB 00:00 2023-06-22T18:04:49.295 INFO:teuthology.orchestra.run.smithi110.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 9.2 MB/s | 225 kB 00:00 2023-06-22T18:04:49.306 INFO:teuthology.orchestra.run.smithi110.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 8.7 MB/s | 126 kB 00:00 2023-06-22T18:04:49.324 INFO:teuthology.orchestra.run.smithi110.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 11 MB/s | 187 kB 00:00 2023-06-22T18:04:49.375 INFO:teuthology.orchestra.run.smithi116.stdout:(5/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 8.8 MB/s | 1.6 MB 00:00 2023-06-22T18:04:49.445 INFO:teuthology.orchestra.run.smithi110.stdout:(106/120): python3-scipy-1.0.0-20.module+el8.1. 48 MB/s | 15 MB 00:00 2023-06-22T18:04:49.449 INFO:teuthology.orchestra.run.smithi110.stdout:(107/120): python3-jinja2-2.10.1-2.el8_0.noarch 4.2 MB/s | 538 kB 00:00 2023-06-22T18:04:49.466 INFO:teuthology.orchestra.run.smithi110.stdout:(108/120): python3-numpy-1.14.3-9.el8.x86_64.rp 22 MB/s | 3.7 MB 00:00 2023-06-22T18:04:49.468 INFO:teuthology.orchestra.run.smithi110.stdout:(109/120): python3-mako-1.0.6-13.el8.noarch.rpm 8.3 MB/s | 157 kB 00:00 2023-06-22T18:04:49.470 INFO:teuthology.orchestra.run.smithi110.stdout:(110/120): socat-1.7.3.3-2.el8.x86_64.rpm 12 MB/s | 302 kB 00:00 2023-06-22T18:04:49.481 INFO:teuthology.orchestra.run.smithi110.stdout:(111/120): python3-pyasn1-modules-0.3.7-6.el8.n 7.3 MB/s | 110 kB 00:00 2023-06-22T18:04:49.483 INFO:teuthology.orchestra.run.smithi110.stdout:(112/120): python3-jsonpointer-1.10-11.el8.noar 1.5 MB/s | 20 kB 00:00 2023-06-22T18:04:49.486 INFO:teuthology.orchestra.run.smithi110.stdout:(113/120): librdkafka-0.11.4-1.el8.x86_64.rpm 19 MB/s | 353 kB 00:00 2023-06-22T18:04:49.498 INFO:teuthology.orchestra.run.smithi110.stdout:(114/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.5 MB/s | 54 kB 00:00 2023-06-22T18:04:49.504 INFO:teuthology.orchestra.run.smithi110.stdout:(115/120): libtool-2.4.6-25.el8.x86_64.rpm 30 MB/s | 709 kB 00:00 2023-06-22T18:04:49.521 INFO:teuthology.orchestra.run.smithi116.stdout:(6/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 4.9 MB/s | 5.2 MB 00:01 2023-06-22T18:04:49.529 INFO:teuthology.orchestra.run.smithi110.stdout:(116/120): automake-1.16.1-7.el8.noarch.rpm 29 MB/s | 713 kB 00:00 2023-06-22T18:04:49.567 INFO:teuthology.orchestra.run.smithi110.stdout:(117/120): openblas-threads-0.3.12-1.el8.x86_64 59 MB/s | 4.7 MB 00:00 2023-06-22T18:04:49.588 INFO:teuthology.orchestra.run.smithi110.stdout:(118/120): openblas-0.3.12-1.el8.x86_64.rpm 52 MB/s | 4.6 MB 00:00 2023-06-22T18:04:49.800 INFO:teuthology.orchestra.run.smithi116.stdout:(7/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 9.3 MB/s | 3.9 MB 00:00 2023-06-22T18:04:50.814 INFO:teuthology.orchestra.run.smithi110.stdout:(119/120): librados2-17.2.6-719.g42ddc46e.el8.x 2.7 MB/s | 3.5 MB 00:01 2023-06-22T18:04:50.933 INFO:teuthology.orchestra.run.smithi110.stdout:(120/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 2.6 MB/s | 3.6 MB 00:01 2023-06-22T18:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-06-22T18:04:50.936 INFO:teuthology.orchestra.run.smithi110.stdout:Total 24 MB/s | 201 MB 00:08 2023-06-22T18:04:50.986 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-06-22T18:04:51.039 INFO:teuthology.orchestra.run.smithi116.stdout:(8/120): ceph-common-17.2.6-719.g42ddc46e.el8.x 9.1 MB/s | 23 MB 00:02 2023-06-22T18:04:51.099 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-06-22T18:04:51.100 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-06-22T18:04:51.114 INFO:teuthology.orchestra.run.smithi116.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 325 kB/s | 24 kB 00:00 2023-06-22T18:04:51.974 INFO:teuthology.orchestra.run.smithi116.stdout:(10/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 7.7 MB/s | 17 MB 00:02 2023-06-22T18:04:52.042 INFO:teuthology.orchestra.run.smithi116.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 3.7 MB/s | 255 kB 00:00 2023-06-22T18:04:52.161 INFO:teuthology.orchestra.run.smithi116.stdout:(12/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 6.3 MB/s | 17 MB 00:02 2023-06-22T18:04:52.161 INFO:teuthology.orchestra.run.smithi116.stdout:(13/120): libcephfs-devel-17.2.6-719.g42ddc46e. 209 kB/s | 25 kB 00:00 2023-06-22T18:04:52.237 INFO:teuthology.orchestra.run.smithi116.stdout:(14/120): libcephsqlite-17.2.6-719.g42ddc46e.el 2.2 MB/s | 174 kB 00:00 2023-06-22T18:04:52.290 INFO:teuthology.orchestra.run.smithi116.stdout:(15/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 6.0 MB/s | 794 kB 00:00 2023-06-22T18:04:52.293 INFO:teuthology.orchestra.run.smithi116.stdout:(16/120): librados-devel-17.2.6-719.g42ddc46e.e 2.3 MB/s | 130 kB 00:00 2023-06-22T18:04:52.335 INFO:teuthology.orchestra.run.smithi116.stdout:(17/120): libradosstriper1-17.2.6-719.g42ddc46e 11 MB/s | 506 kB 00:00 2023-06-22T18:04:52.392 INFO:teuthology.orchestra.run.smithi116.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 787 kB/s | 45 kB 00:00 2023-06-22T18:04:52.452 INFO:teuthology.orchestra.run.smithi116.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 1.9 MB/s | 115 kB 00:00 2023-06-22T18:04:52.518 INFO:teuthology.orchestra.run.smithi116.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 2.8 MB/s | 190 kB 00:00 2023-06-22T18:04:52.592 INFO:teuthology.orchestra.run.smithi116.stdout:(21/120): python3-rados-17.2.6-719.g42ddc46e.el 5.0 MB/s | 371 kB 00:00 2023-06-22T18:04:52.668 INFO:teuthology.orchestra.run.smithi116.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 4.6 MB/s | 357 kB 00:00 2023-06-22T18:04:52.738 INFO:teuthology.orchestra.run.smithi116.stdout:(23/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 1.5 MB/s | 107 kB 00:00 2023-06-22T18:04:52.805 INFO:teuthology.orchestra.run.smithi116.stdout:(24/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.3 MB/s | 91 kB 00:00 2023-06-22T18:04:52.855 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-06-22T18:04:52.860 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-06-22T18:04:52.876 INFO:teuthology.orchestra.run.smithi116.stdout:(25/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 9.5 MB/s | 5.5 MB 00:00 2023-06-22T18:04:52.946 INFO:teuthology.orchestra.run.smithi116.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 2.4 MB/s | 173 kB 00:00 2023-06-22T18:04:53.007 INFO:teuthology.orchestra.run.smithi116.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 375 kB/s | 23 kB 00:00 2023-06-22T18:04:53.075 INFO:teuthology.orchestra.run.smithi116.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 1.7 MB/s | 121 kB 00:00 2023-06-22T18:04:53.310 INFO:teuthology.orchestra.run.smithi116.stdout:(29/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 6.9 MB/s | 3.5 MB 00:00 2023-06-22T18:04:53.315 INFO:teuthology.orchestra.run.smithi116.stdout:(30/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 7.1 MB/s | 1.7 MB 00:00 2023-06-22T18:04:53.390 INFO:teuthology.orchestra.run.smithi116.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 3.1 MB/s | 238 kB 00:00 2023-06-22T18:04:53.455 INFO:teuthology.orchestra.run.smithi116.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 730 kB/s | 47 kB 00:00 2023-06-22T18:04:53.526 INFO:teuthology.orchestra.run.smithi116.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 204 kB/s | 14 kB 00:00 2023-06-22T18:04:53.602 INFO:teuthology.orchestra.run.smithi116.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 3.4 MB/s | 264 kB 00:00 2023-06-22T18:04:53.673 INFO:teuthology.orchestra.run.smithi116.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.1 MB/s | 80 kB 00:00 2023-06-22T18:04:53.747 INFO:teuthology.orchestra.run.smithi116.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 239 kB/s | 18 kB 00:00 2023-06-22T18:04:54.030 INFO:teuthology.orchestra.run.smithi116.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-06-22T18:04:54.140 INFO:teuthology.orchestra.run.smithi116.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 535 kB/s | 59 kB 00:00 2023-06-22T18:04:54.255 INFO:teuthology.orchestra.run.smithi116.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 657 kB/s | 75 kB 00:00 2023-06-22T18:04:54.333 INFO:teuthology.orchestra.run.smithi116.stdout:(40/120): ceph-mgr-diskprediction-local-17.2.6- 7.2 MB/s | 7.4 MB 00:01 2023-06-22T18:04:54.394 INFO:teuthology.orchestra.run.smithi116.stdout:(41/120): python3-asyncssh-2.7.0-2.el8.noarch.r 3.3 MB/s | 476 kB 00:00 2023-06-22T18:04:54.498 INFO:teuthology.orchestra.run.smithi116.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 1.7 MB/s | 185 kB 00:00 2023-06-22T18:04:54.519 INFO:teuthology.orchestra.run.smithi116.stdout:(43/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 235 kB/s | 44 kB 00:00 2023-06-22T18:04:54.546 INFO:teuthology.orchestra.run.smithi116.stdout:(44/120): python3-cachetools-3.1.1-4.el8.noarch 702 kB/s | 33 kB 00:00 2023-06-22T18:04:54.566 INFO:teuthology.orchestra.run.smithi116.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 351 kB/s | 16 kB 00:00 2023-06-22T18:04:54.660 INFO:teuthology.orchestra.run.smithi116.stdout:(46/120): python3-cheroot-8.5.2-1.el8.noarch.rp 1.5 MB/s | 173 kB 00:00 2023-06-22T18:04:54.711 INFO:teuthology.orchestra.run.smithi116.stdout:(47/120): python3-cherrypy-18.4.0-1.el8.noarch. 2.6 MB/s | 384 kB 00:00 2023-06-22T18:04:54.711 INFO:teuthology.orchestra.run.smithi116.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 1.5 MB/s | 81 kB 00:00 2023-06-22T18:04:54.758 INFO:teuthology.orchestra.run.smithi116.stdout:(49/120): python3-jaraco-functools-2.0-4.el8.no 381 kB/s | 18 kB 00:00 2023-06-22T18:04:54.768 INFO:teuthology.orchestra.run.smithi116.stdout:(50/120): python3-jaraco-6.2-6.el8.noarch.rpm 194 kB/s | 11 kB 00:00 2023-06-22T18:04:54.815 INFO:teuthology.orchestra.run.smithi116.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 1.0 MB/s | 49 kB 00:00 2023-06-22T18:04:54.862 INFO:teuthology.orchestra.run.smithi116.stdout:(52/120): python3-kubernetes-11.0.0-6.el8.noarc 11 MB/s | 1.2 MB 00:00 2023-06-22T18:04:54.863 INFO:teuthology.orchestra.run.smithi116.stdout:(53/120): python3-more-itertools-7.2.0-3.el8.no 1.2 MB/s | 59 kB 00:00 2023-06-22T18:04:55.257 INFO:teuthology.orchestra.run.smithi116.stdout:(54/120): ceph-test-17.2.6-719.g42ddc46e.el8.x8 12 MB/s | 52 MB 00:04 2023-06-22T18:04:55.258 INFO:teuthology.orchestra.run.smithi116.stdout:(55/120): python3-natsort-7.1.1-2.el8.noarch.rp 152 kB/s | 60 kB 00:00 2023-06-22T18:04:55.260 INFO:teuthology.orchestra.run.smithi116.stdout:(56/120): python3-pecan-1.3.2-9.el8.noarch.rpm 713 kB/s | 283 kB 00:00 2023-06-22T18:04:55.305 INFO:teuthology.orchestra.run.smithi116.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 725 kB/s | 34 kB 00:00 2023-06-22T18:04:55.352 INFO:teuthology.orchestra.run.smithi116.stdout:(58/120): python3-rsa-4.8-1.el8.noarch.rpm 1.3 MB/s | 61 kB 00:00 2023-06-22T18:04:55.353 INFO:teuthology.orchestra.run.smithi116.stdout:(59/120): python3-routes-2.4.1-12.el8.noarch.rp 2.1 MB/s | 196 kB 00:00 2023-06-22T18:04:55.396 INFO:teuthology.orchestra.run.smithi116.stdout:(60/120): python3-portend-2.6-1.el8.noarch.rpm 112 kB/s | 16 kB 00:00 2023-06-22T18:04:55.398 INFO:teuthology.orchestra.run.smithi116.stdout:(61/120): python3-simplegeneric-0.8.1-17.el8.no 408 kB/s | 19 kB 00:00 2023-06-22T18:04:55.400 INFO:teuthology.orchestra.run.smithi116.stdout:(62/120): python3-singledispatch-3.4.0.3-18.el8 514 kB/s | 24 kB 00:00 2023-06-22T18:04:55.443 INFO:teuthology.orchestra.run.smithi116.stdout:(63/120): python3-tempora-1.14.1-5.el8.noarch.r 630 kB/s | 29 kB 00:00 2023-06-22T18:04:55.444 INFO:teuthology.orchestra.run.smithi116.stdout:(64/120): python3-trustme-0.6.0-4.el8.noarch.rp 599 kB/s | 27 kB 00:00 2023-06-22T18:04:55.449 INFO:teuthology.orchestra.run.smithi116.stdout:(65/120): python3-waitress-1.4.3-1.el8.noarch.r 5.0 MB/s | 255 kB 00:00 2023-06-22T18:04:55.491 INFO:teuthology.orchestra.run.smithi116.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 1.3 MB/s | 61 kB 00:00 2023-06-22T18:04:55.497 INFO:teuthology.orchestra.run.smithi116.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 1.8 MB/s | 85 kB 00:00 2023-06-22T18:04:55.547 INFO:teuthology.orchestra.run.smithi116.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 416 kB/s | 23 kB 00:00 2023-06-22T18:04:55.552 INFO:teuthology.orchestra.run.smithi116.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 14 MB/s | 70 kB 00:00 2023-06-22T18:04:55.584 INFO:teuthology.orchestra.run.smithi116.stdout:(70/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 596 kB/s | 19 kB 00:00 2023-06-22T18:04:55.598 INFO:teuthology.orchestra.run.smithi116.stdout:(71/120): python3-jwt-1.6.1-2.el8.noarch.rpm 2.8 MB/s | 43 kB 00:00 2023-06-22T18:04:55.611 INFO:teuthology.orchestra.run.smithi116.stdout:(72/120): python3-webob-1.8.5-1.el8.1.noarch.rp 1.5 MB/s | 251 kB 00:00 2023-06-22T18:04:55.614 INFO:teuthology.orchestra.run.smithi116.stdout:(73/120): python3-oauthlib-2.1.0-1.el8.noarch.r 9.6 MB/s | 155 kB 00:00 2023-06-22T18:04:55.630 INFO:teuthology.orchestra.run.smithi116.stdout:(74/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 3.1 MB/s | 48 kB 00:00 2023-06-22T18:04:55.643 INFO:teuthology.orchestra.run.smithi116.stdout:(75/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 6.8 MB/s | 220 kB 00:00 2023-06-22T18:04:55.646 INFO:teuthology.orchestra.run.smithi116.stdout:(76/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 14 MB/s | 229 kB 00:00 2023-06-22T18:04:55.657 INFO:teuthology.orchestra.run.smithi116.stdout:(77/120): python3-requests-oauthlib-1.0.0-1.el8 3.1 MB/s | 43 kB 00:00 2023-06-22T18:04:55.671 INFO:teuthology.orchestra.run.smithi116.stdout:(78/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 1.3 MB/s | 18 kB 00:00 2023-06-22T18:04:55.691 INFO:teuthology.orchestra.run.smithi116.stdout:(79/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 53 MB/s | 2.3 MB 00:00 2023-06-22T18:04:55.698 INFO:teuthology.orchestra.run.smithi116.stdout:(80/120): pcre2-devel-10.32-2.el8.x86_64.rpm 22 MB/s | 605 kB 00:00 2023-06-22T18:04:55.706 INFO:teuthology.orchestra.run.smithi116.stdout:(81/120): fuse-2.9.7-12.el8.x86_64.rpm 5.6 MB/s | 83 kB 00:00 2023-06-22T18:04:55.713 INFO:teuthology.orchestra.run.smithi116.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 3.8 MB/s | 59 kB 00:00 2023-06-22T18:04:55.722 INFO:teuthology.orchestra.run.smithi116.stdout:(83/120): libquadmath-8.4.1-1.el8.x86_64.rpm 10 MB/s | 169 kB 00:00 2023-06-22T18:04:55.728 INFO:teuthology.orchestra.run.smithi116.stdout:(84/120): libcom_err-devel-1.45.6-1.el8.x86_64. 2.6 MB/s | 38 kB 00:00 2023-06-22T18:04:55.737 INFO:teuthology.orchestra.run.smithi116.stdout:(85/120): libsepol-devel-2.9-2.el8.x86_64.rpm 5.5 MB/s | 86 kB 00:00 2023-06-22T18:04:55.742 INFO:teuthology.orchestra.run.smithi116.stdout:(86/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 4.1 MB/s | 58 kB 00:00 2023-06-22T18:04:55.751 INFO:teuthology.orchestra.run.smithi116.stdout:(87/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.5 MB/s | 47 kB 00:00 2023-06-22T18:04:55.761 INFO:teuthology.orchestra.run.smithi116.stdout:(88/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 29 MB/s | 559 kB 00:00 2023-06-22T18:04:55.770 INFO:teuthology.orchestra.run.smithi116.stdout:(89/120): libgfortran-8.4.1-1.el8.x86_64.rpm 33 MB/s | 639 kB 00:00 2023-06-22T18:04:55.777 INFO:teuthology.orchestra.run.smithi116.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 4.5 MB/s | 70 kB 00:00 2023-06-22T18:04:55.788 INFO:teuthology.orchestra.run.smithi116.stdout:(91/120): libselinux-devel-2.9-5.el8.x86_64.rpm 11 MB/s | 200 kB 00:00 2023-06-22T18:04:55.794 INFO:teuthology.orchestra.run.smithi116.stdout:(92/120): thrift-0.13.0-2.el8.x86_64.rpm 5.8 MB/s | 1.7 MB 00:00 2023-06-22T18:04:55.796 INFO:teuthology.orchestra.run.smithi116.stdout:(93/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 4.9 MB/s | 97 kB 00:00 2023-06-22T18:04:55.811 INFO:teuthology.orchestra.run.smithi116.stdout:(94/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.8 MB/s | 27 kB 00:00 2023-06-22T18:04:55.825 INFO:teuthology.orchestra.run.smithi116.stdout:(95/120): python3-prettytable-0.7.2-14.el8.noar 1.4 MB/s | 44 kB 00:00 2023-06-22T18:04:55.828 INFO:teuthology.orchestra.run.smithi116.stdout:(96/120): jq-1.5-12.el8.x86_64.rpm 9.4 MB/s | 161 kB 00:00 2023-06-22T18:04:55.843 INFO:teuthology.orchestra.run.smithi116.stdout:(97/120): flex-2.6.1-9.el8.x86_64.rpm 18 MB/s | 320 kB 00:00 2023-06-22T18:04:55.849 INFO:teuthology.orchestra.run.smithi116.stdout:(98/120): python3-werkzeug-0.12.2-4.el8.noarch. 22 MB/s | 457 kB 00:00 2023-06-22T18:04:55.862 INFO:teuthology.orchestra.run.smithi116.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 2.8 MB/s | 39 kB 00:00 2023-06-22T18:04:55.886 INFO:teuthology.orchestra.run.smithi116.stdout:(100/120): bison-3.0.4-10.el8.x86_64.rpm 29 MB/s | 688 kB 00:00 2023-06-22T18:04:55.902 INFO:teuthology.orchestra.run.smithi116.stdout:(101/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 15 MB/s | 259 kB 00:00 2023-06-22T18:04:55.930 INFO:teuthology.orchestra.run.smithi116.stdout:(102/120): python3-babel-2.5.1-5.el8.noarch.rpm 55 MB/s | 4.8 MB 00:00 2023-06-22T18:04:55.931 INFO:teuthology.orchestra.run.smithi116.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 7.7 MB/s | 225 kB 00:00 2023-06-22T18:04:55.945 INFO:teuthology.orchestra.run.smithi116.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 8.1 MB/s | 126 kB 00:00 2023-06-22T18:04:55.961 INFO:teuthology.orchestra.run.smithi116.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 12 MB/s | 187 kB 00:00 2023-06-22T18:04:55.982 INFO:teuthology.orchestra.run.smithi116.stdout:(106/120): python3-jinja2-2.10.1-2.el8_0.noarch 26 MB/s | 538 kB 00:00 2023-06-22T18:04:56.086 INFO:teuthology.orchestra.run.smithi116.stdout:(107/120): python3-scipy-1.0.0-20.module+el8.1. 49 MB/s | 15 MB 00:00 2023-06-22T18:04:56.097 INFO:teuthology.orchestra.run.smithi116.stdout:(108/120): python3-numpy-1.14.3-9.el8.x86_64.rp 23 MB/s | 3.7 MB 00:00 2023-06-22T18:04:56.100 INFO:teuthology.orchestra.run.smithi116.stdout:(109/120): socat-1.7.3.3-2.el8.x86_64.rpm 2.5 MB/s | 302 kB 00:00 2023-06-22T18:04:56.101 INFO:teuthology.orchestra.run.smithi116.stdout:(110/120): python3-mako-1.0.6-13.el8.noarch.rpm 10 MB/s | 157 kB 00:00 2023-06-22T18:04:56.113 INFO:teuthology.orchestra.run.smithi116.stdout:(111/120): python3-pyasn1-modules-0.3.7-6.el8.n 6.9 MB/s | 110 kB 00:00 2023-06-22T18:04:56.126 INFO:teuthology.orchestra.run.smithi116.stdout:(112/120): librdkafka-0.11.4-1.el8.x86_64.rpm 13 MB/s | 353 kB 00:00 2023-06-22T18:04:56.127 INFO:teuthology.orchestra.run.smithi116.stdout:(113/120): python3-jsonpointer-1.10-11.el8.noar 784 kB/s | 20 kB 00:00 2023-06-22T18:04:56.134 INFO:teuthology.orchestra.run.smithi116.stdout:(114/120): libtool-2.4.6-25.el8.x86_64.rpm 33 MB/s | 709 kB 00:00 2023-06-22T18:04:56.143 INFO:teuthology.orchestra.run.smithi116.stdout:(115/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.2 MB/s | 54 kB 00:00 2023-06-22T18:04:56.171 INFO:teuthology.orchestra.run.smithi116.stdout:(116/120): automake-1.16.1-7.el8.noarch.rpm 25 MB/s | 713 kB 00:00 2023-06-22T18:04:56.207 INFO:teuthology.orchestra.run.smithi116.stdout:(117/120): openblas-threads-0.3.12-1.el8.x86_64 59 MB/s | 4.7 MB 00:00 2023-06-22T18:04:56.228 INFO:teuthology.orchestra.run.smithi116.stdout:(118/120): openblas-0.3.12-1.el8.x86_64.rpm 49 MB/s | 4.6 MB 00:00 2023-06-22T18:04:57.553 INFO:teuthology.orchestra.run.smithi116.stdout:(119/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 2.7 MB/s | 3.6 MB 00:01 2023-06-22T18:04:57.723 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-06-22T18:04:57.991 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-22T18:04:58.173 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-22T18:04:58.333 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-22T18:04:58.564 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-22T18:04:58.942 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-22T18:04:59.206 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-22T18:04:59.372 INFO:teuthology.orchestra.run.smithi116.stdout:(120/120): librados2-17.2.6-719.g42ddc46e.el8.x 1.1 MB/s | 3.5 MB 00:03 2023-06-22T18:04:59.376 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2023-06-22T18:04:59.377 INFO:teuthology.orchestra.run.smithi116.stdout:Total 18 MB/s | 201 MB 00:11 2023-06-22T18:04:59.429 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2023-06-22T18:04:59.545 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2023-06-22T18:04:59.545 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2023-06-22T18:04:59.609 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-22T18:04:59.840 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-22T18:04:59.975 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-22T18:05:01.006 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T18:05:01.108 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T18:05:01.228 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2023-06-22T18:05:01.232 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2023-06-22T18:05:01.312 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T18:05:01.646 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-06-22T18:05:02.323 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-22T18:05:02.522 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T18:05:02.639 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T18:05:02.910 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-22T18:05:03.121 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-22T18:05:03.331 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T18:05:03.439 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T18:05:03.614 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-22T18:05:03.692 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-22T18:05:03.860 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-06-22T18:05:04.136 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-06-22T18:05:04.396 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-06-22T18:05:04.584 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-06-22T18:05:04.731 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-22T18:05:04.906 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-22T18:05:05.072 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-22T18:05:05.655 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-22T18:05:05.855 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-22T18:05:05.893 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2023-06-22T18:05:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-22T18:05:06.336 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-22T18:05:06.386 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-22T18:05:06.495 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-22T18:05:06.686 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-22T18:05:06.897 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-06-22T18:05:07.075 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-22T18:05:07.369 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-22T18:05:07.641 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-06-22T18:05:07.775 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-22T18:05:07.917 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-06-22T18:05:08.011 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-22T18:05:08.131 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-22T18:05:08.215 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-06-22T18:05:08.352 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T18:05:08.443 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T18:05:08.492 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-22T18:05:08.666 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T18:05:08.695 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-22T18:05:08.826 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-22T18:05:09.001 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-06-22T18:05:09.113 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-22T18:05:09.369 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-06-22T18:05:09.647 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-06-22T18:05:09.687 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-22T18:05:09.877 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T18:05:09.880 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-22T18:05:09.992 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T18:05:10.256 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-22T18:05:10.417 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-22T18:05:10.644 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T18:05:10.759 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T18:05:10.927 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-22T18:05:11.004 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-22T18:05:11.156 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-06-22T18:05:11.441 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-06-22T18:05:11.700 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-06-22T18:05:11.847 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-06-22T18:05:11.966 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-22T18:05:12.027 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-22T18:05:12.166 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-22T18:05:12.310 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-22T18:05:12.414 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-06-22T18:05:13.048 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-22T18:05:13.285 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-22T18:05:13.470 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-06-22T18:05:13.822 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-22T18:05:13.826 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-06-22T18:05:14.035 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-06-22T18:05:14.118 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-06-22T18:05:14.394 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-22T18:05:14.641 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-22T18:05:14.763 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-06-22T18:05:14.880 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-22T18:05:15.021 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-06-22T18:05:15.093 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-22T18:05:15.277 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-06-22T18:05:15.364 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-22T18:05:15.530 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-22T18:05:15.570 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-06-22T18:05:15.675 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-22T18:05:15.761 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-06-22T18:05:15.808 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-22T18:05:15.962 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-06-22T18:05:16.058 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-22T18:05:16.191 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-06-22T18:05:16.316 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-06-22T18:05:16.404 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-22T18:05:16.485 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-22T18:05:16.559 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-06-22T18:05:16.687 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-06-22T18:05:16.776 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-22T18:05:16.849 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-06-22T18:05:17.061 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-06-22T18:05:17.250 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-06-22T18:05:17.423 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-06-22T18:05:17.606 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-06-22T18:05:17.732 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-06-22T18:05:17.961 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-06-22T18:05:18.098 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-06-22T18:05:18.428 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-06-22T18:05:18.709 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-06-22T18:05:18.821 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-22T18:05:18.997 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-06-22T18:05:19.149 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-06-22T18:05:19.296 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-06-22T18:05:19.496 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-22T18:05:19.659 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-22T18:05:19.919 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-22T18:05:20.157 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-06-22T18:05:20.162 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-22T18:05:20.377 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-22T18:05:20.515 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-06-22T18:05:20.541 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-22T18:05:20.723 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-22T18:05:20.788 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-06-22T18:05:20.913 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-22T18:05:21.024 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-22T18:05:21.158 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-22T18:05:21.220 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-22T18:05:21.406 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-22T18:05:21.444 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-22T18:05:21.623 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-22T18:05:21.631 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-22T18:05:21.785 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-22T18:05:21.963 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-22T18:05:22.040 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-22T18:05:22.216 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-06-22T18:05:22.317 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-22T18:05:22.399 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-06-22T18:05:22.552 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-22T18:05:22.615 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-06-22T18:05:22.824 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-06-22T18:05:22.837 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-22T18:05:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-22T18:05:23.050 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-22T18:05:23.095 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-22T18:05:23.234 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-06-22T18:05:23.321 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-22T18:05:23.392 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-06-22T18:05:23.549 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-06-22T18:05:23.594 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-22T18:05:23.721 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-06-22T18:05:23.892 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-06-22T18:05:24.102 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-06-22T18:05:24.262 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-06-22T18:05:24.489 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-06-22T18:05:24.667 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-06-22T18:05:24.746 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-22T18:05:24.947 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-06-22T18:05:25.076 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-22T18:05:25.230 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-06-22T18:05:25.288 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-22T18:05:25.498 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-06-22T18:05:25.610 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-22T18:05:25.782 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-06-22T18:05:25.873 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-22T18:05:25.933 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-22T18:05:26.105 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-22T18:05:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-22T18:05:26.371 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-22T18:05:26.438 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-22T18:05:26.617 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-22T18:05:26.654 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-22T18:05:26.790 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-22T18:05:26.870 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-22T18:05:26.984 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-22T18:05:27.127 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-22T18:05:27.333 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-22T18:05:27.527 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-22T18:05:27.752 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-22T18:05:27.936 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-22T18:05:28.114 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-22T18:05:28.339 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-22T18:05:28.611 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-22T18:05:28.839 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-22T18:05:29.112 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-22T18:05:29.340 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-22T18:05:29.548 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-22T18:05:29.772 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-22T18:05:30.927 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-22T18:05:31.256 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-22T18:05:31.517 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-22T18:05:31.824 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-22T18:05:32.017 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T18:05:32.093 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-22T18:05:32.244 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T18:05:32.431 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-22T18:05:32.683 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-22T18:05:32.879 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-22T18:05:33.073 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T18:05:33.108 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-22T18:05:33.701 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-22T18:05:33.830 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-22T18:05:33.969 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-22T18:05:38.050 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T18:05:38.282 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T18:05:38.890 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-22T18:05:39.184 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-22T18:05:39.275 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-22T18:05:39.415 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-22T18:05:52.010 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-22T18:05:52.011 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2023-06-22T18:05:52.011 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2023-06-22T18:05:52.011 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /dev 2023-06-22T18:05:52.011 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /run 2023-06-22T18:05:52.011 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2023-06-22T18:05:52.011 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2023-06-22T18:05:52.012 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2023-06-22T18:05:52.012 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2023-06-22T18:05:52.012 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /dev 2023-06-22T18:05:52.012 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:05:52.416 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-22T18:05:53.903 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-22T18:05:54.065 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-22T18:05:54.288 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-22T18:05:54.425 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-22T18:05:54.815 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-22T18:05:54.966 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-22T18:05:54.983 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-22T18:05:57.134 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-22T18:05:57.135 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2023-06-22T18:05:57.135 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2023-06-22T18:05:57.135 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /dev 2023-06-22T18:05:57.135 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /run 2023-06-22T18:05:57.135 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2023-06-22T18:05:57.135 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2023-06-22T18:05:57.135 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2023-06-22T18:05:57.136 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2023-06-22T18:05:57.136 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /dev 2023-06-22T18:05:57.136 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:05:57.545 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-22T18:05:57.707 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-22T18:05:57.870 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-22T18:05:58.072 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-22T18:05:58.291 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-22T18:05:58.515 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-22T18:05:58.733 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-22T18:05:58.939 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-22T18:05:59.024 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-22T18:05:59.181 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-22T18:05:59.403 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-22T18:05:59.562 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-22T18:05:59.910 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-22T18:05:59.952 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-22T18:06:00.124 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-22T18:06:00.145 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-22T18:06:00.162 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-22T18:06:00.226 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-22T18:06:00.612 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-22T18:06:00.814 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-22T18:06:00.852 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-22T18:06:01.073 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-22T18:06:01.084 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-22T18:06:02.680 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-22T18:06:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-22T18:06:02.914 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-22T18:06:03.010 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-22T18:06:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-22T18:06:03.184 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-22T18:06:03.190 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-22T18:06:03.234 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-22T18:06:03.370 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-22T18:06:03.390 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-22T18:06:03.544 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-22T18:06:03.739 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-22T18:06:03.759 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-22T18:06:03.889 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-22T18:06:03.955 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-22T18:06:04.926 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-22T18:06:05.069 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-22T18:06:05.168 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-22T18:06:05.569 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-22T18:06:05.748 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-22T18:06:05.785 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-22T18:06:05.960 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-22T18:06:05.972 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-22T18:06:07.541 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-22T18:06:07.780 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-22T18:06:07.872 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-22T18:06:08.010 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-22T18:06:08.054 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-22T18:06:08.202 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-22T18:06:08.550 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-22T18:06:08.710 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-22T18:06:08.824 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-22T18:06:09.804 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-22T18:06:10.221 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-06-22T18:06:10.393 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 110/122 2023-06-22T18:06:10.659 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-06-22T18:06:10.855 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-06-22T18:06:10.867 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-06-22T18:06:11.018 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-22T18:06:11.185 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-22T18:06:11.339 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-22T18:06:11.541 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-22T18:06:11.828 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-22T18:06:12.046 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-06-22T18:06:12.100 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-06-22T18:06:12.247 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-06-22T18:06:12.255 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-06-22T18:06:12.424 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-06-22T18:06:12.424 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-22T18:06:12.658 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-22T18:06:12.658 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T18:06:13.220 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T18:06:13.221 INFO:teuthology.orchestra.run.smithi110.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-22T18:06:13.221 INFO:teuthology.orchestra.run.smithi110.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-22T18:06:13.221 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:13.570 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-22T18:06:13.571 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-22T18:06:13.571 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-22T18:06:13.571 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-22T18:06:13.571 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-22T18:06:13.571 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T18:06:13.571 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T18:06:13.572 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-22T18:06:13.572 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-22T18:06:13.572 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T18:06:13.572 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-22T18:06:13.572 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-22T18:06:13.572 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T18:06:13.573 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-22T18:06:13.573 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-22T18:06:13.573 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-22T18:06:13.573 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-22T18:06:13.573 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-22T18:06:13.573 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-22T18:06:13.574 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-22T18:06:13.574 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-22T18:06:13.574 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-22T18:06:13.574 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-22T18:06:13.574 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-22T18:06:13.574 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-22T18:06:13.581 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-22T18:06:13.581 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-22T18:06:13.582 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-22T18:06:13.582 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-22T18:06:13.582 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-22T18:06:13.582 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-22T18:06:13.582 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-22T18:06:13.582 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-22T18:06:13.582 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-22T18:06:13.583 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-22T18:06:13.583 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-22T18:06:13.583 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-22T18:06:13.583 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-22T18:06:13.583 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-22T18:06:13.583 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-22T18:06:13.584 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-22T18:06:13.584 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-22T18:06:13.584 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-22T18:06:13.584 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-22T18:06:13.584 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-22T18:06:13.584 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-22T18:06:13.585 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-22T18:06:13.585 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-22T18:06:13.585 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-22T18:06:13.585 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-22T18:06:13.585 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-22T18:06:13.586 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-22T18:06:13.586 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-22T18:06:13.586 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-22T18:06:13.586 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-22T18:06:13.586 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-22T18:06:13.586 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-22T18:06:13.587 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-22T18:06:13.587 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-22T18:06:13.587 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-22T18:06:13.587 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-22T18:06:13.587 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-22T18:06:13.587 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-22T18:06:13.587 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-22T18:06:13.588 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-22T18:06:13.588 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-22T18:06:13.588 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-22T18:06:13.588 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-22T18:06:13.588 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-22T18:06:13.588 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-22T18:06:13.589 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-22T18:06:13.589 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-06-22T18:06:13.589 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-06-22T18:06:13.589 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-06-22T18:06:13.589 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-06-22T18:06:13.589 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-06-22T18:06:13.590 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-06-22T18:06:13.590 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-06-22T18:06:13.590 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-06-22T18:06:13.590 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-06-22T18:06:13.590 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-06-22T18:06:13.590 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-06-22T18:06:13.590 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-06-22T18:06:13.591 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-06-22T18:06:13.591 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-06-22T18:06:13.591 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-06-22T18:06:13.591 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-06-22T18:06:13.591 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-06-22T18:06:13.591 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-06-22T18:06:13.591 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-06-22T18:06:13.592 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-06-22T18:06:13.592 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-06-22T18:06:13.592 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-06-22T18:06:13.592 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-06-22T18:06:13.592 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-06-22T18:06:13.592 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-06-22T18:06:13.592 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-06-22T18:06:13.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-06-22T18:06:13.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-06-22T18:06:13.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-06-22T18:06:13.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-06-22T18:06:13.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-06-22T18:06:13.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-06-22T18:06:13.593 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-06-22T18:06:13.594 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-06-22T18:06:13.594 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-06-22T18:06:13.594 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-06-22T18:06:13.594 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-06-22T18:06:13.594 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-06-22T18:06:13.594 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-06-22T18:06:13.594 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-06-22T18:06:13.594 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-06-22T18:06:13.595 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-06-22T18:06:13.595 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-06-22T18:06:13.595 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-06-22T18:06:13.595 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-06-22T18:06:13.595 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-06-22T18:06:13.595 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-06-22T18:06:13.596 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-22T18:06:13.596 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-22T18:06:13.596 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-22T18:06:13.643 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-22T18:06:14.606 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-22T18:06:14.984 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-06-22T18:06:15.187 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 110/122 2023-06-22T18:06:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-06-22T18:06:15.576 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-06-22T18:06:15.587 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-06-22T18:06:15.730 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-22T18:06:15.828 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T18:06:15.829 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:42 ago on Thu 22 Jun 2023 06:04:33 PM UTC. 2023-06-22T18:06:15.905 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-22T18:06:16.042 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-22T18:06:16.252 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-22T18:06:16.484 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-22T18:06:16.700 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-06-22T18:06:16.753 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-06-22T18:06:16.950 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-06-22T18:06:16.959 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-06-22T18:06:17.077 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-06-22T18:06:17.078 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-22T18:06:17.320 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-22T18:06:17.320 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T18:06:17.858 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T18:06:17.858 INFO:teuthology.orchestra.run.smithi116.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-22T18:06:17.859 INFO:teuthology.orchestra.run.smithi116.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-22T18:06:17.859 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:18.160 INFO:teuthology.orchestra.run.smithi110.stdout:Installed products updated. 2023-06-22T18:06:18.188 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-22T18:06:18.188 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-22T18:06:18.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-22T18:06:18.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-22T18:06:18.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-22T18:06:18.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-22T18:06:18.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-22T18:06:18.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-22T18:06:18.189 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-22T18:06:18.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-22T18:06:18.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-22T18:06:18.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-22T18:06:18.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-22T18:06:18.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-22T18:06:18.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-22T18:06:18.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-22T18:06:18.190 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-22T18:06:18.191 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-22T18:06:18.191 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-22T18:06:18.191 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-22T18:06:18.191 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-22T18:06:18.191 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-22T18:06:18.191 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-22T18:06:18.191 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-22T18:06:18.192 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-22T18:06:18.192 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-22T18:06:18.192 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-22T18:06:18.198 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-22T18:06:18.198 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-22T18:06:18.198 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-22T18:06:18.198 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-22T18:06:18.199 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-22T18:06:18.199 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-22T18:06:18.199 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-22T18:06:18.199 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-22T18:06:18.199 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-22T18:06:18.199 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-22T18:06:18.199 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-22T18:06:18.199 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-22T18:06:18.200 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-22T18:06:18.200 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-22T18:06:18.200 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-22T18:06:18.200 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-22T18:06:18.200 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-22T18:06:18.200 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-22T18:06:18.200 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-22T18:06:18.200 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-22T18:06:18.201 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-22T18:06:18.201 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-22T18:06:18.201 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-22T18:06:18.201 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-22T18:06:18.201 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-22T18:06:18.201 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-22T18:06:18.201 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-22T18:06:18.202 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-22T18:06:18.202 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-22T18:06:18.202 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-22T18:06:18.202 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-22T18:06:18.202 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-22T18:06:18.202 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-22T18:06:18.202 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-22T18:06:18.202 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-22T18:06:18.203 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-22T18:06:18.203 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-22T18:06:18.203 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-22T18:06:18.203 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-22T18:06:18.203 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-22T18:06:18.203 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-22T18:06:18.203 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-22T18:06:18.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-22T18:06:18.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-22T18:06:18.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-06-22T18:06:18.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-06-22T18:06:18.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-06-22T18:06:18.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-06-22T18:06:18.204 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-06-22T18:06:18.205 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-06-22T18:06:18.205 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-06-22T18:06:18.205 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-06-22T18:06:18.205 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-06-22T18:06:18.205 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-06-22T18:06:18.205 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-06-22T18:06:18.205 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-06-22T18:06:18.205 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-06-22T18:06:18.206 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-06-22T18:06:18.206 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-06-22T18:06:18.206 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-06-22T18:06:18.206 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-06-22T18:06:18.206 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-06-22T18:06:18.206 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-06-22T18:06:18.206 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-06-22T18:06:18.206 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-06-22T18:06:18.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-06-22T18:06:18.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-06-22T18:06:18.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-06-22T18:06:18.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-06-22T18:06:18.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-06-22T18:06:18.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-06-22T18:06:18.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-06-22T18:06:18.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-06-22T18:06:18.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-06-22T18:06:18.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-06-22T18:06:18.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-06-22T18:06:18.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-06-22T18:06:18.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-06-22T18:06:18.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-06-22T18:06:18.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-06-22T18:06:18.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-06-22T18:06:18.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-06-22T18:06:18.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-06-22T18:06:18.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-06-22T18:06:18.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-06-22T18:06:18.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-06-22T18:06:18.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-06-22T18:06:18.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-06-22T18:06:18.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-06-22T18:06:18.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-06-22T18:06:18.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-06-22T18:06:18.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-22T18:06:18.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-22T18:06:18.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-22T18:06:18.289 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:18.289 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2023-06-22T18:06:18.289 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.290 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.290 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-06-22T18:06:18.290 INFO:teuthology.orchestra.run.smithi110.stdout: automake-1.16.1-7.el8.noarch 2023-06-22T18:06:18.290 INFO:teuthology.orchestra.run.smithi110.stdout: bison-3.0.4-10.el8.x86_64 2023-06-22T18:06:18.290 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.290 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.290 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.291 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.291 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:18.291 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.291 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.291 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.291 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:18.291 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:18.291 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:18.292 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:18.292 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:18.292 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.292 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.292 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:18.292 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.292 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.293 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.293 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:18.293 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:18.293 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.293 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:18.293 INFO:teuthology.orchestra.run.smithi110.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-22T18:06:18.293 INFO:teuthology.orchestra.run.smithi110.stdout: flex-2.6.1-9.el8.x86_64 2023-06-22T18:06:18.293 INFO:teuthology.orchestra.run.smithi110.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-22T18:06:18.294 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-22T18:06:18.294 INFO:teuthology.orchestra.run.smithi110.stdout: jq-1.5-12.el8.x86_64 2023-06-22T18:06:18.294 INFO:teuthology.orchestra.run.smithi110.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-22T18:06:18.294 INFO:teuthology.orchestra.run.smithi110.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-22T18:06:18.294 INFO:teuthology.orchestra.run.smithi110.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-22T18:06:18.294 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.294 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.295 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.295 INFO:teuthology.orchestra.run.smithi110.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-22T18:06:18.295 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-22T18:06:18.295 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-22T18:06:18.295 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-22T18:06:18.295 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-22T18:06:18.295 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.296 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.296 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-22T18:06:18.296 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.296 INFO:teuthology.orchestra.run.smithi110.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-22T18:06:18.296 INFO:teuthology.orchestra.run.smithi110.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-22T18:06:18.296 INFO:teuthology.orchestra.run.smithi110.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-22T18:06:18.296 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-22T18:06:18.296 INFO:teuthology.orchestra.run.smithi110.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-22T18:06:18.297 INFO:teuthology.orchestra.run.smithi110.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-22T18:06:18.297 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-22T18:06:18.297 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-22T18:06:18.297 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-22T18:06:18.297 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-22T18:06:18.297 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-22T18:06:18.297 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-22T18:06:18.298 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-22T18:06:18.298 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-22T18:06:18.298 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-22T18:06:18.298 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-22T18:06:18.298 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-22T18:06:18.298 INFO:teuthology.orchestra.run.smithi110.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-22T18:06:18.298 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-22T18:06:18.298 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.299 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.299 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.299 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-22T18:06:18.299 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-22T18:06:18.299 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-22T18:06:18.299 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-22T18:06:18.299 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-22T18:06:18.299 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-22T18:06:18.300 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-22T18:06:18.300 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-22T18:06:18.300 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-22T18:06:18.300 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-22T18:06:18.300 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-22T18:06:18.300 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-22T18:06:18.300 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-22T18:06:18.300 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-22T18:06:18.301 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-22T18:06:18.301 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-22T18:06:18.301 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-22T18:06:18.301 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-22T18:06:18.301 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-22T18:06:18.301 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-22T18:06:18.301 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-2.6-1.el8.noarch 2023-06-22T18:06:18.301 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-22T18:06:18.302 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-22T18:06:18.302 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-22T18:06:18.302 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-22T18:06:18.302 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.302 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.302 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-22T18:06:18.302 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-22T18:06:18.303 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.303 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-22T18:06:18.303 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-22T18:06:18.303 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-22T18:06:18.303 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-22T18:06:18.303 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-22T18:06:18.303 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-22T18:06:18.303 INFO:teuthology.orchestra.run.smithi110.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-22T18:06:18.304 INFO:teuthology.orchestra.run.smithi110.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-22T18:06:18.304 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-22T18:06:18.304 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-22T18:06:18.304 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-22T18:06:18.304 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-22T18:06:18.304 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-22T18:06:18.304 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.305 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.305 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:18.305 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-22T18:06:18.305 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-22T18:06:18.305 INFO:teuthology.orchestra.run.smithi110.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-22T18:06:18.305 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-22T18:06:18.306 INFO:teuthology.orchestra.run.smithi110.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-22T18:06:18.306 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:18.306 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-06-22T18:06:18.554 DEBUG:teuthology.parallel:result is None 2023-06-22T18:06:20.428 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-22T18:06:20.428 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:41 ago on Thu 22 Jun 2023 06:04:39 PM UTC. 2023-06-22T18:06:22.836 INFO:teuthology.orchestra.run.smithi116.stdout:Installed products updated. 2023-06-22T18:06:22.964 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:22.964 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2023-06-22T18:06:22.964 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.965 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.965 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2023-06-22T18:06:22.965 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.1-7.el8.noarch 2023-06-22T18:06:22.965 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.0.4-10.el8.x86_64 2023-06-22T18:06:22.965 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.965 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.965 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:22.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:22.966 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:22.967 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:22.967 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:22.967 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:22.967 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.967 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.967 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:22.967 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-22T18:06:22.968 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-22T18:06:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.1-9.el8.x86_64 2023-06-22T18:06:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-22T18:06:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-22T18:06:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.5-12.el8.x86_64 2023-06-22T18:06:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-22T18:06:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-22T18:06:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-22T18:06:22.969 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-22T18:06:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-22T18:06:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-22T18:06:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-22T18:06:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-22T18:06:22.970 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-22T18:06:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-22T18:06:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-22T18:06:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-22T18:06:22.971 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-22T18:06:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-22T18:06:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-22T18:06:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-22T18:06:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-22T18:06:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-22T18:06:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-22T18:06:22.972 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-22T18:06:22.973 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-22T18:06:22.973 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-22T18:06:22.973 INFO:teuthology.orchestra.run.smithi116.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-22T18:06:22.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-22T18:06:22.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-22T18:06:22.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-22T18:06:22.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-22T18:06:22.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-22T18:06:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-22T18:06:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-22T18:06:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-22T18:06:22.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-22T18:06:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-22T18:06:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-22T18:06:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-22T18:06:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-22T18:06:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-22T18:06:22.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-22T18:06:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-22T18:06:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-22T18:06:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-22T18:06:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-22T18:06:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-22T18:06:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-22T18:06:22.976 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-22T18:06:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-22T18:06:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-22T18:06:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-22T18:06:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-2.6-1.el8.noarch 2023-06-22T18:06:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-22T18:06:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-22T18:06:22.977 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-22T18:06:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-22T18:06:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-22T18:06:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-22T18:06:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-22T18:06:22.978 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-22T18:06:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-22T18:06:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-22T18:06:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-22T18:06:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-22T18:06:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-22T18:06:22.979 INFO:teuthology.orchestra.run.smithi116.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-22T18:06:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-22T18:06:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-22T18:06:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-22T18:06:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-22T18:06:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-22T18:06:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.980 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-22T18:06:22.981 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-22T18:06:22.981 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-22T18:06:22.981 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-22T18:06:22.981 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-22T18:06:22.981 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-22T18:06:22.982 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:22.982 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2023-06-22T18:06:23.174 DEBUG:teuthology.parallel:result is None 2023-06-22T18:06:23.175 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T18:06:23.319 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-22T18:06:23.339 INFO:teuthology.orchestra.run.smithi110.stdout:17.2.6-719.g42ddc46e.el8 2023-06-22T18:06:23.340 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-22T18:06:23.340 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-22T18:06:23.341 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-22T18:06:23.483 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-22T18:06:23.504 INFO:teuthology.orchestra.run.smithi116.stdout:17.2.6-719.g42ddc46e.el8 2023-06-22T18:06:23.504 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-22T18:06:23.505 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-22T18:06:23.505 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-06-22T18:06:23.506 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:23.506 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-22T18:06:23.545 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:23.545 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-22T18:06:23.584 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-06-22T18:06:23.585 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:23.585 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2023-06-22T18:06:23.610 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-22T18:06:23.676 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:23.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2023-06-22T18:06:23.701 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-22T18:06:23.765 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-06-22T18:06:23.765 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:23.766 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-22T18:06:23.789 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-22T18:06:23.854 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:23.854 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-22T18:06:23.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-22T18:06:23.942 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-06-22T18:06:23.942 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:23.943 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2023-06-22T18:06:23.967 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-22T18:06:24.032 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:24.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2023-06-22T18:06:24.058 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-22T18:06:24.122 INFO:teuthology.run_tasks:Running task ceph... 2023-06-22T18:06:24.208 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-06-22T18:06:24.208 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2023-06-22T18:06:24.210 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2023-06-22T18:06:24.234 INFO:tasks.ceph:Disabling ceph logrotate... 2023-06-22T18:06:24.234 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-22T18:06:24.274 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-22T18:06:24.298 INFO:tasks.ceph:Creating extra log directories... 2023-06-22T18:06:24.298 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-22T18:06:24.337 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-22T18:06:24.367 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-06-22T18:06:24.367 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-06-22T18:06:24.368 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-06-22T18:06:24.368 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:24.368 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-22T18:06:24.414 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-22T18:06:24.530 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-22T18:06:24.556 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:24.556 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-22T18:06:24.570 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-22T18:06:24.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-22T18:06:24.714 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-06-22T18:06:24.714 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-06-22T18:06:24.715 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857', 'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'description': 'fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7311857', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 9037, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'targets': {'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE8qRhuRUXj1iFiQd/Gr7ahxAb/xrAR5aYNjNZNzG8lND1ManSzV5hIdYStVClXN02MXcaxubtpjWEjQm1azrps=', 'smithi116.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOlY0LD4+AaVWVxqWRbI1ZOcEjeJnJyA/uEqjVQqVqKkgaSUHAk8XiDdozZLzJuZTuX+5Sc911PDBBIpIciEguKTBN1jMRTf4XX0SDPQbjZfukPSSEBZGxWzuVyfPk1UlCGdlHbkGHH0xwvWBerXFjA7qSH4rXOR7NizG70cnbCdih0S/hzVNGJDLVSvJ5gAiqE6PxjQ7Zf3zLNiwzvMhD4dolr7HwhSnAOddGaqSo8ZU+6VM+BYWFy2syLN7tDV1f8UCYt26ItcV6MMvwQRI4usWpNssQnI8J2j2Dn2PmPBdNFnmwFp4UyKK7Aq0lT032ygSvVdD7Oa5ZKlFg1UaZcNXfAZudG/qXwN0rzebd4cvKVq0Hw3jpHiTLqYGqQtE2Cz1iFmmoD7LSoVBXGItMBZJw+L+UhlkM0+wXAk/44WSV0CRMOcR8aDx5NiCnHv5Vw3NED+SpkjGsLzFTwwWQQoWIf39TilTSGqpZrWnp9ulvrgIv7VAVEAD6fiaVPqU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '076bbebc42a14f7d568aaa78eabb0038327bcb23', 'timestamp': '2023-06-22_15:25:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-06-22T18:06:24.716 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-22T18:06:24.717 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-22T18:06:24.764 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2023-06-22T18:06:24.772 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2023-06-22T18:06:24.828 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:24.828 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2023-06-22T18:06:24.848 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-22T18:06:24.848 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2023-06-22T18:06:24.904 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-22T18:06:24.904 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T18:06:24.904 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 207651 Links: 1 2023-06-22T18:06:24.905 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T18:06:24.905 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T18:06:24.905 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-06-22 18:06:13.134575297 +0000 2023-06-22T18:06:24.905 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-06-22 18:02:47.667008996 +0000 2023-06-22T18:06:24.905 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-06-22 18:02:47.667008996 +0000 2023-06-22T18:06:24.905 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-06-22T18:06:24.906 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-22T18:06:24.967 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-06-22T18:06:24.967 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-06-22T18:06:24.968 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.8361e-05 s, 5.2 MB/s 2023-06-22T18:06:24.968 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-22T18:06:25.026 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2023-06-22T18:06:25.081 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-22T18:06:25.082 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T18:06:25.082 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 214326 Links: 1 2023-06-22T18:06:25.082 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T18:06:25.082 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T18:06:25.082 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-06-22 18:06:13.134575297 +0000 2023-06-22T18:06:25.083 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-06-22 18:02:48.176000304 +0000 2023-06-22T18:06:25.083 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-06-22 18:02:48.176000304 +0000 2023-06-22T18:06:25.083 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-06-22T18:06:25.083 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-22T18:06:25.146 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-06-22T18:06:25.146 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-06-22T18:06:25.146 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.3129e-05 s, 5.5 MB/s 2023-06-22T18:06:25.147 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-22T18:06:25.202 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2023-06-22T18:06:25.257 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-22T18:06:25.258 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T18:06:25.258 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 211775 Links: 1 2023-06-22T18:06:25.258 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T18:06:25.258 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T18:06:25.259 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-06-22 18:06:13.134575297 +0000 2023-06-22T18:06:25.259 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-06-22 18:02:48.708991201 +0000 2023-06-22T18:06:25.259 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-06-22 18:02:48.708991201 +0000 2023-06-22T18:06:25.260 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-06-22T18:06:25.260 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-22T18:06:25.320 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-06-22T18:06:25.320 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-06-22T18:06:25.321 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.2778e-05 s, 5.5 MB/s 2023-06-22T18:06:25.321 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-22T18:06:25.378 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2023-06-22T18:06:25.433 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-22T18:06:25.434 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T18:06:25.434 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 215196 Links: 1 2023-06-22T18:06:25.434 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T18:06:25.434 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T18:06:25.434 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-06-22 18:06:13.134575297 +0000 2023-06-22T18:06:25.434 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-06-22 18:02:49.209982647 +0000 2023-06-22T18:06:25.435 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-06-22 18:02:49.209982647 +0000 2023-06-22T18:06:25.435 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-06-22T18:06:25.435 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-22T18:06:25.495 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-06-22T18:06:25.495 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-06-22T18:06:25.495 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.465e-05 s, 5.4 MB/s 2023-06-22T18:06:25.496 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-22T18:06:25.551 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-22T18:06:25.551 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:25.552 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2023-06-22T18:06:25.566 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-22T18:06:25.566 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2023-06-22T18:06:25.620 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-22T18:06:25.620 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T18:06:25.621 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 209497 Links: 1 2023-06-22T18:06:25.621 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T18:06:25.621 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T18:06:25.621 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-06-22 18:06:17.772344846 +0000 2023-06-22T18:06:25.621 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-06-22 17:58:18.459143579 +0000 2023-06-22T18:06:25.621 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-06-22 17:58:18.459143579 +0000 2023-06-22T18:06:25.621 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-06-22T18:06:25.622 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-22T18:06:25.684 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-06-22T18:06:25.684 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-06-22T18:06:25.685 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000307258 s, 1.7 MB/s 2023-06-22T18:06:25.685 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-22T18:06:25.741 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2023-06-22T18:06:25.795 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-22T18:06:25.796 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T18:06:25.796 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 213230 Links: 1 2023-06-22T18:06:25.796 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T18:06:25.796 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T18:06:25.796 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-06-22 18:06:17.772344846 +0000 2023-06-22T18:06:25.796 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-06-22 17:58:18.951135521 +0000 2023-06-22T18:06:25.796 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-06-22 17:58:18.951135521 +0000 2023-06-22T18:06:25.797 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-06-22T18:06:25.797 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-22T18:06:25.859 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-06-22T18:06:25.859 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-06-22T18:06:25.859 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000215805 s, 2.4 MB/s 2023-06-22T18:06:25.860 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-22T18:06:25.917 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2023-06-22T18:06:25.971 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-22T18:06:25.972 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T18:06:25.972 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 209605 Links: 1 2023-06-22T18:06:25.972 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T18:06:25.972 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T18:06:25.972 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-06-22 18:06:17.773344830 +0000 2023-06-22T18:06:25.973 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-06-22 17:58:19.442127487 +0000 2023-06-22T18:06:25.973 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-06-22 17:58:19.442127487 +0000 2023-06-22T18:06:25.973 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-06-22T18:06:25.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-22T18:06:26.034 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-06-22T18:06:26.034 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-06-22T18:06:26.035 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000237816 s, 2.2 MB/s 2023-06-22T18:06:26.035 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-22T18:06:26.091 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2023-06-22T18:06:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-22T18:06:26.146 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-22T18:06:26.146 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 6h/6d Inode: 214389 Links: 1 2023-06-22T18:06:26.147 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-22T18:06:26.147 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2023-06-22T18:06:26.147 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2023-06-22 18:06:17.773344830 +0000 2023-06-22T18:06:26.148 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2023-06-22 17:58:19.951119159 +0000 2023-06-22T18:06:26.148 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2023-06-22 17:58:19.951119159 +0000 2023-06-22T18:06:26.148 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2023-06-22T18:06:26.148 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-22T18:06:26.208 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2023-06-22T18:06:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2023-06-22T18:06:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000180509 s, 2.8 MB/s 2023-06-22T18:06:26.210 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-22T18:06:26.266 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-22T18:06:26.266 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-22T18:06:26.266 INFO:tasks.ceph:Generating config... 2023-06-22T18:06:26.267 INFO:tasks.ceph:[client] client mount timeout = 600 2023-06-22T18:06:26.268 INFO:tasks.ceph:[client] debug client = 20 2023-06-22T18:06:26.268 INFO:tasks.ceph:[client] debug ms = 1 2023-06-22T18:06:26.268 INFO:tasks.ceph:[client] pid file = 2023-06-22T18:06:26.268 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-06-22T18:06:26.269 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-06-22T18:06:26.269 INFO:tasks.ceph:[mds] debug mds = 20 2023-06-22T18:06:26.269 INFO:tasks.ceph:[mds] debug ms = 1 2023-06-22T18:06:26.269 INFO:tasks.ceph:[mds] mds debug frag = True 2023-06-22T18:06:26.269 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-06-22T18:06:26.270 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-06-22T18:06:26.270 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-06-22T18:06:26.270 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-06-22T18:06:26.270 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-06-22T18:06:26.270 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-06-22T18:06:26.270 INFO:tasks.ceph:[mgr] debug client = 20 2023-06-22T18:06:26.271 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-06-22T18:06:26.271 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-06-22T18:06:26.271 INFO:tasks.ceph:[mgr] debug ms = 1 2023-06-22T18:06:26.271 INFO:tasks.ceph:[mon] debug mon = 20 2023-06-22T18:06:26.271 INFO:tasks.ceph:[mon] debug ms = 1 2023-06-22T18:06:26.272 INFO:tasks.ceph:[mon] debug paxos = 20 2023-06-22T18:06:26.272 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-06-22T18:06:26.272 INFO:tasks.ceph:[osd] bdev async discard = True 2023-06-22T18:06:26.272 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-06-22T18:06:26.272 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-06-22T18:06:26.273 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-06-22T18:06:26.273 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-06-22T18:06:26.273 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-06-22T18:06:26.273 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-06-22T18:06:26.273 INFO:tasks.ceph:[osd] debug ms = 1 2023-06-22T18:06:26.274 INFO:tasks.ceph:[osd] debug osd = 20 2023-06-22T18:06:26.274 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-06-22T18:06:26.274 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-06-22T18:06:26.274 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-06-22T18:06:26.274 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-06-22T18:06:26.275 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-06-22T18:06:26.275 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-06-22T18:06:26.275 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-06-22T18:06:26.275 INFO:tasks.ceph:Setting up mon.a... 2023-06-22T18:06:26.275 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-06-22T18:06:26.331 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.keyring 2023-06-22T18:06:26.333 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-06-22T18:06:26.372 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-22T18:06:26.436 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.110'), ('mon.b', '172.21.15.116'), ('mon.c', '[v2:172.21.15.116:3301,v1:172.21.15.116:6790]')] 2023-06-22T18:06:26.436 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.110,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-06-22T18:06:26.437 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:26.438 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-06-22T18:06:26.490 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.110 --add b 172.21.15.116 --addv c '[v2:172.21.15.116:3301,v1:172.21.15.116:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:26.572 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:26.572 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: generated fsid cbed9da7-33df-4858-8860-5c7cacb062c0 2023-06-22T18:06:26.573 INFO:teuthology.orchestra.run.smithi110.stdout:setting min_mon_release = octopus 2023-06-22T18:06:26.573 INFO:teuthology.orchestra.run.smithi110.stdout:epoch 0 2023-06-22T18:06:26.573 INFO:teuthology.orchestra.run.smithi110.stdout:fsid cbed9da7-33df-4858-8860-5c7cacb062c0 2023-06-22T18:06:26.573 INFO:teuthology.orchestra.run.smithi110.stdout:last_changed 2023-06-22T18:06:26.569635+0000 2023-06-22T18:06:26.574 INFO:teuthology.orchestra.run.smithi110.stdout:created 2023-06-22T18:06:26.569635+0000 2023-06-22T18:06:26.574 INFO:teuthology.orchestra.run.smithi110.stdout:min_mon_release 15 (octopus) 2023-06-22T18:06:26.574 INFO:teuthology.orchestra.run.smithi110.stdout:election_strategy: 1 2023-06-22T18:06:26.574 INFO:teuthology.orchestra.run.smithi110.stdout:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2023-06-22T18:06:26.575 INFO:teuthology.orchestra.run.smithi110.stdout:1: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.b 2023-06-22T18:06:26.575 INFO:teuthology.orchestra.run.smithi110.stdout:2: [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] mon.c 2023-06-22T18:06:26.575 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-06-22T18:06:26.577 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-06-22T18:06:26.632 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cbed9da7-33df-4858-8860-5c7cacb062c0... 2023-06-22T18:06:26.634 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-22T18:06:26.674 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-22T18:06:26.716 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2023-06-22T18:06:26.716 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2023-06-22T18:06:26.717 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-22T18:06:26.717 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2023-06-22T18:06:26.717 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.717 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2023-06-22T18:06:26.717 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.717 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2023-06-22T18:06:26.717 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.718 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2023-06-22T18:06:26.718 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2023-06-22T18:06:26.718 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.718 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2023-06-22T18:06:26.718 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2023-06-22T18:06:26.718 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.718 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-22T18:06:26.719 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2023-06-22T18:06:26.719 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.719 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2023-06-22T18:06:26.719 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-22T18:06:26.719 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.719 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2023-06-22T18:06:26.719 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.719 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2023-06-22T18:06:26.720 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2023-06-22T18:06:26.720 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2023-06-22T18:06:26.720 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2023-06-22T18:06:26.720 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2023-06-22T18:06:26.720 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2023-06-22T18:06:26.720 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2023-06-22T18:06:26.721 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-22T18:06:26.721 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-22T18:06:26.721 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2023-06-22T18:06:26.721 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.721 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-22T18:06:26.721 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.722 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2023-06-22T18:06:26.722 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.722 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2023-06-22T18:06:26.722 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.722 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2023-06-22T18:06:26.722 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2023-06-22T18:06:26.722 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2023-06-22T18:06:26.722 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "172.21.15.110,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2023-06-22T18:06:26.723 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = cbed9da7-33df-4858-8860-5c7cacb062c0 2023-06-22T18:06:26.723 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.723 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2023-06-22T18:06:26.723 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2023-06-22T18:06:26.723 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.723 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2023-06-22T18:06:26.723 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2023-06-22T18:06:26.723 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2023-06-22T18:06:26.724 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.724 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2023-06-22T18:06:26.724 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2023-06-22T18:06:26.724 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.724 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2023-06-22T18:06:26.724 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2023-06-22T18:06:26.724 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2023-06-22T18:06:26.724 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.725 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2023-06-22T18:06:26.725 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2023-06-22T18:06:26.725 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.725 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2023-06-22T18:06:26.725 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.725 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2023-06-22T18:06:26.725 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.725 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2023-06-22T18:06:26.726 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.726 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2023-06-22T18:06:26.726 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2023-06-22T18:06:26.726 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.726 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2023-06-22T18:06:26.726 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2023-06-22T18:06:26.726 INFO:teuthology.orchestra.run.smithi110.stdout: bdev async discard = True 2023-06-22T18:06:26.726 INFO:teuthology.orchestra.run.smithi110.stdout: bdev enable discard = True 2023-06-22T18:06:26.727 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore allocator = bitmap 2023-06-22T18:06:26.727 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2023-06-22T18:06:26.727 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2023-06-22T18:06:26.727 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 1/20 2023-06-22T18:06:26.727 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 1/20 2023-06-22T18:06:26.727 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-06-22T18:06:26.727 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2023-06-22T18:06:26.728 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 4/10 2023-06-22T18:06:26.728 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2023-06-22T18:06:26.728 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2023-06-22T18:06:26.728 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2023-06-22T18:06:26.728 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2023-06-22T18:06:26.728 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2023-06-22T18:06:26.728 INFO:teuthology.orchestra.run.smithi110.stdout: osd op complaint time = 180 2023-06-22T18:06:26.728 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.729 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2023-06-22T18:06:26.729 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-06-22T18:06:26.729 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2023-06-22T18:06:26.729 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-06-22T18:06:26.729 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-06-22T18:06:26.729 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-06-22T18:06:26.729 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-06-22T18:06:26.729 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2023-06-22T18:06:26.730 INFO:teuthology.orchestra.run.smithi110.stdout: debug client = 20 2023-06-22T18:06:26.730 INFO:teuthology.orchestra.run.smithi110.stdout: debug finisher = 20 2023-06-22T18:06:26.730 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.730 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2023-06-22T18:06:26.730 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-06-22T18:06:26.730 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-06-22T18:06:26.730 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2023-06-22T18:06:26.730 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-06-22T18:06:26.731 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2023-06-22T18:06:26.731 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2023-06-22T18:06:26.731 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-06-22T18:06:26.731 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2023-06-22T18:06:26.731 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2023-06-22T18:06:26.731 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-06-22T18:06:26.731 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.731 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-22T18:06:26.732 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2023-06-22T18:06:26.732 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2023-06-22T18:06:26.732 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.732 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2023-06-22T18:06:26.732 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-22T18:06:26.732 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-22T18:06:26.732 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.732 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2023-06-22T18:06:26.733 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2023-06-22T18:06:26.733 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.733 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2023-06-22T18:06:26.733 INFO:teuthology.orchestra.run.smithi110.stdout: mon op complaint time = 120 2023-06-22T18:06:26.733 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:06:26.733 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2023-06-22T18:06:26.734 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2023-06-22T18:06:26.734 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2023-06-22T18:06:26.734 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2023-06-22T18:06:26.734 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-22T18:06:26.734 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-22T18:06:26.734 INFO:teuthology.orchestra.run.smithi110.stdout: client mount timeout = 600 2023-06-22T18:06:26.734 INFO:teuthology.orchestra.run.smithi110.stdout: debug client = 20 2023-06-22T18:06:26.734 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-06-22T18:06:26.735 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = "" 2023-06-22T18:06:26.735 INFO:teuthology.orchestra.run.smithi110.stdout: rados mon op timeout = 900 2023-06-22T18:06:26.735 INFO:teuthology.orchestra.run.smithi110.stdout: rados osd op timeout = 900 2023-06-22T18:06:26.735 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2023-06-22T18:06:26.735 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2023-06-22T18:06:26.735 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2023-06-22T18:06:26.735 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.a] 2023-06-22T18:06:26.735 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.c] 2023-06-22T18:06:26.736 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.b] 2023-06-22T18:06:26.736 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.d] 2023-06-22T18:06:26.736 INFO:teuthology.orchestra.run.smithi110.stdout:[mds] 2023-06-22T18:06:26.736 INFO:teuthology.orchestra.run.smithi110.stdout: debug mds = 20 2023-06-22T18:06:26.736 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-06-22T18:06:26.736 INFO:teuthology.orchestra.run.smithi110.stdout: mds debug frag = True 2023-06-22T18:06:26.736 INFO:teuthology.orchestra.run.smithi110.stdout: mds debug scatterstat = True 2023-06-22T18:06:26.736 INFO:teuthology.orchestra.run.smithi110.stdout: mds op complaint time = 180 2023-06-22T18:06:26.737 INFO:teuthology.orchestra.run.smithi110.stdout: mds verify scatter = True 2023-06-22T18:06:26.737 INFO:teuthology.orchestra.run.smithi110.stdout: osd op complaint time = 180 2023-06-22T18:06:26.737 INFO:teuthology.orchestra.run.smithi110.stdout: rados mon op timeout = 900 2023-06-22T18:06:26.737 INFO:teuthology.orchestra.run.smithi110.stdout: rados osd op timeout = 900 2023-06-22T18:06:26.739 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2023-06-22T18:06:26.739 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2023-06-22T18:06:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-22T18:06:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2023-06-22T18:06:26.740 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2023-06-22T18:06:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2023-06-22T18:06:26.741 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2023-06-22T18:06:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2023-06-22T18:06:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.742 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2023-06-22T18:06:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2023-06-22T18:06:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-22T18:06:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2023-06-22T18:06:26.743 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2023-06-22T18:06:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-22T18:06:26.744 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.745 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2023-06-22T18:06:26.745 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.745 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2023-06-22T18:06:26.745 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2023-06-22T18:06:26.746 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2023-06-22T18:06:26.746 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2023-06-22T18:06:26.746 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2023-06-22T18:06:26.746 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2023-06-22T18:06:26.746 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2023-06-22T18:06:26.746 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-22T18:06:26.747 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-22T18:06:26.747 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2023-06-22T18:06:26.747 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.747 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-22T18:06:26.747 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.747 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2023-06-22T18:06:26.747 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.747 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2023-06-22T18:06:26.748 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.748 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2023-06-22T18:06:26.748 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2023-06-22T18:06:26.748 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2023-06-22T18:06:26.748 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.110,172.21.15.116,[v2:172.21.15.116:3301,v1:172.21.15.116:6790]" 2023-06-22T18:06:26.748 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = cbed9da7-33df-4858-8860-5c7cacb062c0 2023-06-22T18:06:26.748 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.749 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2023-06-22T18:06:26.749 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2023-06-22T18:06:26.749 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.749 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2023-06-22T18:06:26.749 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2023-06-22T18:06:26.749 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2023-06-22T18:06:26.749 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.750 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2023-06-22T18:06:26.750 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2023-06-22T18:06:26.750 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.750 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2023-06-22T18:06:26.750 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2023-06-22T18:06:26.750 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2023-06-22T18:06:26.750 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.750 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2023-06-22T18:06:26.751 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2023-06-22T18:06:26.751 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.751 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2023-06-22T18:06:26.751 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.751 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2023-06-22T18:06:26.751 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.751 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2023-06-22T18:06:26.751 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.752 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2023-06-22T18:06:26.752 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2023-06-22T18:06:26.752 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.752 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2023-06-22T18:06:26.752 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2023-06-22T18:06:26.752 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2023-06-22T18:06:26.752 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2023-06-22T18:06:26.753 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2023-06-22T18:06:26.753 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2023-06-22T18:06:26.753 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2023-06-22T18:06:26.753 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2023-06-22T18:06:26.753 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2023-06-22T18:06:26.753 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-06-22T18:06:26.754 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2023-06-22T18:06:26.754 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2023-06-22T18:06:26.754 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2023-06-22T18:06:26.754 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2023-06-22T18:06:26.754 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2023-06-22T18:06:26.754 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2023-06-22T18:06:26.754 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2023-06-22T18:06:26.754 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2023-06-22T18:06:26.755 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.755 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2023-06-22T18:06:26.755 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-06-22T18:06:26.755 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2023-06-22T18:06:26.755 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2023-06-22T18:06:26.755 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2023-06-22T18:06:26.755 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2023-06-22T18:06:26.756 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2023-06-22T18:06:26.756 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2023-06-22T18:06:26.756 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2023-06-22T18:06:26.756 INFO:teuthology.orchestra.run.smithi116.stdout: debug finisher = 20 2023-06-22T18:06:26.756 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.756 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2023-06-22T18:06:26.756 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-06-22T18:06:26.756 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2023-06-22T18:06:26.757 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2023-06-22T18:06:26.757 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2023-06-22T18:06:26.757 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2023-06-22T18:06:26.757 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2023-06-22T18:06:26.757 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2023-06-22T18:06:26.757 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2023-06-22T18:06:26.758 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2023-06-22T18:06:26.758 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2023-06-22T18:06:26.758 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.758 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-22T18:06:26.758 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2023-06-22T18:06:26.758 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2023-06-22T18:06:26.758 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.759 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2023-06-22T18:06:26.759 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-22T18:06:26.759 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-22T18:06:26.759 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.759 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2023-06-22T18:06:26.759 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2023-06-22T18:06:26.759 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.760 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2023-06-22T18:06:26.760 INFO:teuthology.orchestra.run.smithi116.stdout: mon op complaint time = 120 2023-06-22T18:06:26.760 INFO:teuthology.orchestra.run.smithi116.stdout: 2023-06-22T18:06:26.760 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2023-06-22T18:06:26.760 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2023-06-22T18:06:26.760 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2023-06-22T18:06:26.760 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2023-06-22T18:06:26.760 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-22T18:06:26.761 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-22T18:06:26.761 INFO:teuthology.orchestra.run.smithi116.stdout: client mount timeout = 600 2023-06-22T18:06:26.761 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2023-06-22T18:06:26.761 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-06-22T18:06:26.761 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = "" 2023-06-22T18:06:26.762 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2023-06-22T18:06:26.762 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2023-06-22T18:06:26.762 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2023-06-22T18:06:26.762 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2023-06-22T18:06:26.763 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2023-06-22T18:06:26.763 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2023-06-22T18:06:26.763 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2023-06-22T18:06:26.763 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2023-06-22T18:06:26.763 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2023-06-22T18:06:26.764 INFO:teuthology.orchestra.run.smithi116.stdout:[mds] 2023-06-22T18:06:26.764 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds = 20 2023-06-22T18:06:26.764 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2023-06-22T18:06:26.764 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug frag = True 2023-06-22T18:06:26.764 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug scatterstat = True 2023-06-22T18:06:26.765 INFO:teuthology.orchestra.run.smithi116.stdout: mds op complaint time = 180 2023-06-22T18:06:26.765 INFO:teuthology.orchestra.run.smithi116.stdout: mds verify scatter = True 2023-06-22T18:06:26.765 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2023-06-22T18:06:26.765 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2023-06-22T18:06:26.765 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2023-06-22T18:06:26.766 INFO:tasks.ceph:Creating admin key on mon.a... 2023-06-22T18:06:26.766 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-06-22T18:06:27.044 INFO:tasks.ceph:Copying monmap to all nodes... 2023-06-22T18:06:27.045 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:27.045 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-06-22T18:06:27.059 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:27.060 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-06-22T18:06:27.113 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2023-06-22T18:06:27.114 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:27.115 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-22T18:06:27.115 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-22T18:06:27.234 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:27.235 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:27.262 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2023-06-22T18:06:27.263 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:27.263 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-22T18:06:27.263 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-22T18:06:27.297 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:27.297 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:27.351 INFO:tasks.ceph:Setting up mon nodes... 2023-06-22T18:06:27.352 INFO:tasks.ceph:Setting up mgr nodes... 2023-06-22T18:06:27.352 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-06-22T18:06:27.400 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-06-22T18:06:27.402 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-06-22T18:06:27.461 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-06-22T18:06:27.463 INFO:tasks.ceph:Setting up mds nodes... 2023-06-22T18:06:27.464 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-06-22T18:06:27.511 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-06-22T18:06:27.513 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-06-22T18:06:27.537 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-06-22T18:06:27.626 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-06-22T18:06:27.628 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-06-22T18:06:27.653 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-06-22T18:06:27.705 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-06-22T18:06:27.708 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-06-22T18:06:27.733 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-06-22T18:06:27.821 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-06-22T18:06:27.823 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-06-22T18:06:27.848 INFO:tasks.ceph_client:Setting up client nodes... 2023-06-22T18:06:27.848 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-06-22T18:06:27.886 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-06-22T18:06:27.899 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-06-22T18:06:27.977 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-06-22T18:06:27.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-06-22T18:06:28.028 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-06-22T18:06:28.042 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-06-22T18:06:28.119 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-06-22T18:06:28.133 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-06-22T18:06:28.134 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-22T18:06:28.134 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-06-22T18:06:28.159 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-22T18:06:28.159 INFO:tasks.ceph:role: osd.0 2023-06-22T18:06:28.159 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2023-06-22T18:06:28.160 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-22T18:06:28.223 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T18:06:28.223 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T18:06:28.224 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T18:06:28.224 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-06-22T18:06:28.224 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T18:06:28.224 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-06-22T18:06:28.224 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T18:06:28.224 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T18:06:28.224 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T18:06:28.225 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T18:06:28.246 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-06-22T18:06:28.248 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-06-22T18:06:28.248 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-06-22T18:06:28.327 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-06-22T18:06:28.396 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-06-22T18:06:28.461 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-22T18:06:28.462 INFO:tasks.ceph:role: osd.1 2023-06-22T18:06:28.462 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2023-06-22T18:06:28.462 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-22T18:06:28.524 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T18:06:28.525 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T18:06:28.525 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T18:06:28.525 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-06-22T18:06:28.525 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T18:06:28.525 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-06-22T18:06:28.526 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T18:06:28.526 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T18:06:28.526 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T18:06:28.526 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T18:06:28.547 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-06-22T18:06:28.548 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-06-22T18:06:28.549 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-06-22T18:06:28.625 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-06-22T18:06:28.695 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-06-22T18:06:28.762 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-22T18:06:28.762 INFO:tasks.ceph:role: osd.2 2023-06-22T18:06:28.762 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2023-06-22T18:06:28.763 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-22T18:06:28.827 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T18:06:28.827 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T18:06:28.828 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T18:06:28.828 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-06-22T18:06:28.828 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T18:06:28.828 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-06-22T18:06:28.828 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T18:06:28.829 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T18:06:28.829 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T18:06:28.829 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T18:06:28.849 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-06-22T18:06:28.851 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-06-22T18:06:28.851 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-06-22T18:06:28.928 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-06-22T18:06:28.997 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-06-22T18:06:29.063 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-22T18:06:29.063 INFO:tasks.ceph:role: osd.3 2023-06-22T18:06:29.063 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com 2023-06-22T18:06:29.064 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-22T18:06:29.126 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T18:06:29.127 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T18:06:29.127 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T18:06:29.127 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 2023-06-22T18:06:29.127 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T18:06:29.127 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-06-22T18:06:29.127 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T18:06:29.128 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T18:06:29.128 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T18:06:29.128 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T18:06:29.149 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-06-22T18:06:29.150 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-06-22T18:06:29.150 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-06-22T18:06:29.227 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-06-22T18:06:29.295 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:29.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:29.392+0000 7f5e53d84540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-06-22T18:06:29.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:29.393+0000 7f5e53d84540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-06-22T18:06:29.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:29.393+0000 7f5e53d84540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-22T18:06:29.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:29.393+0000 7f5e53d84540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-22T18:06:29.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:29.393+0000 7f5e53d84540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-22T18:06:29.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:29.393+0000 7f5e53d84540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-06-22T18:06:29.918 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:29.914+0000 7f5e53d84540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:30.687 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:30.684+0000 7f5e53d84540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:31.460 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:31.457+0000 7f5e53d84540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:32.295 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-06-22T18:06:32.319 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:32.395+0000 7f8e756fb540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-06-22T18:06:32.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:32.395+0000 7f8e756fb540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-06-22T18:06:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:32.395+0000 7f8e756fb540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-22T18:06:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:32.395+0000 7f8e756fb540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-22T18:06:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:32.395+0000 7f8e756fb540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-22T18:06:32.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:32.396+0000 7f8e756fb540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-06-22T18:06:32.927 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:32.924+0000 7f8e756fb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:33.696 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:33.692+0000 7f8e756fb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:34.461 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:34.457+0000 7f8e756fb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:35.271 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-06-22T18:06:35.336 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:35.413+0000 7fd6e81a6540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-06-22T18:06:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:35.413+0000 7fd6e81a6540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-06-22T18:06:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:35.413+0000 7fd6e81a6540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-22T18:06:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:35.413+0000 7fd6e81a6540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-22T18:06:35.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:35.413+0000 7fd6e81a6540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-22T18:06:35.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:35.413+0000 7fd6e81a6540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-06-22T18:06:35.946 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:35.942+0000 7fd6e81a6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:36.723 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:36.719+0000 7fd6e81a6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:37.494 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:37.491+0000 7fd6e81a6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:38.290 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-06-22T18:06:38.355 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:38.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:38.432+0000 7f3181f96540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-06-22T18:06:38.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:38.432+0000 7f3181f96540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-06-22T18:06:38.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:38.432+0000 7f3181f96540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-22T18:06:38.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:38.432+0000 7f3181f96540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-22T18:06:38.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:38.432+0000 7f3181f96540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-22T18:06:38.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:38.433+0000 7f3181f96540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-06-22T18:06:38.970 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:38.967+0000 7f3181f96540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:39.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:39.743+0000 7f3181f96540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:40.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:40.517+0000 7f3181f96540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:41.327 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-06-22T18:06:41.396 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-06-22T18:06:41.424 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-22T18:06:41.424 INFO:tasks.ceph:role: osd.4 2023-06-22T18:06:41.424 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2023-06-22T18:06:41.424 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-22T18:06:41.495 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T18:06:41.496 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T18:06:41.496 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T18:06:41.496 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 2023-06-22T18:06:41.496 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T18:06:41.496 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-06-22T18:06:41.496 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T18:06:41.496 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T18:06:41.497 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T18:06:41.497 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T18:06:41.921 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-06-22T18:06:41.923 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-06-22T18:06:41.923 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-06-22T18:06:42.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-06-22T18:06:42.121 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-06-22T18:06:42.186 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-22T18:06:42.186 INFO:tasks.ceph:role: osd.5 2023-06-22T18:06:42.187 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2023-06-22T18:06:42.187 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-22T18:06:42.259 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T18:06:42.260 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T18:06:42.260 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T18:06:42.260 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 2023-06-22T18:06:42.260 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T18:06:42.260 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-06-22T18:06:42.260 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T18:06:42.261 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T18:06:42.261 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T18:06:42.261 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T18:06:42.678 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-06-22T18:06:42.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-06-22T18:06:42.680 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-06-22T18:06:42.786 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-06-22T18:06:42.813 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-06-22T18:06:42.878 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-22T18:06:42.878 INFO:tasks.ceph:role: osd.6 2023-06-22T18:06:42.878 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com 2023-06-22T18:06:42.878 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-22T18:06:42.950 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T18:06:42.950 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T18:06:42.950 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T18:06:42.950 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 2023-06-22T18:06:42.950 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T18:06:42.951 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-06-22T18:06:42.951 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T18:06:42.951 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T18:06:42.951 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T18:06:42.951 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T18:06:43.410 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-06-22T18:06:43.411 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-06-22T18:06:43.412 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-06-22T18:06:43.573 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-06-22T18:06:43.601 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-06-22T18:06:43.666 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-22T18:06:43.666 INFO:tasks.ceph:role: osd.7 2023-06-22T18:06:43.667 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com 2023-06-22T18:06:43.667 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-22T18:06:43.737 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-22T18:06:43.738 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-22T18:06:43.738 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-22T18:06:43.738 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 2023-06-22T18:06:43.738 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-22T18:06:43.738 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2023-06-22T18:06:43.738 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-22T18:06:43.739 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-22T18:06:43.739 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-22T18:06:43.739 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-22T18:06:44.168 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2023-06-22T18:06:44.169 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2023-06-22T18:06:44.170 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-06-22T18:06:44.344 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-06-22T18:06:44.375 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:44.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:44.496+0000 7fcecc54a540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-06-22T18:06:44.501 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:44.496+0000 7fcecc54a540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-06-22T18:06:44.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:44.496+0000 7fcecc54a540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-22T18:06:44.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:44.496+0000 7fcecc54a540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-22T18:06:44.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:44.496+0000 7fcecc54a540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-22T18:06:44.502 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:44.497+0000 7fcecc54a540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-06-22T18:06:45.032 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:45.028+0000 7fcecc54a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:45.807 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:45.803+0000 7fcecc54a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:46.585 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:46.581+0000 7fcecc54a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:47.393 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-06-22T18:06:47.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:47.492+0000 7fcf3c204540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-06-22T18:06:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:47.493+0000 7fcf3c204540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-06-22T18:06:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:47.493+0000 7fcf3c204540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-22T18:06:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:47.493+0000 7fcf3c204540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-22T18:06:47.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:47.493+0000 7fcf3c204540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-22T18:06:47.498 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:47.493+0000 7fcf3c204540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-06-22T18:06:48.031 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:48.027+0000 7fcf3c204540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:48.810 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:48.806+0000 7fcf3c204540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:49.572+0000 7fcf3c204540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:50.367 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-06-22T18:06:50.435 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:50.514+0000 7f1953909540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-06-22T18:06:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:50.514+0000 7f1953909540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-06-22T18:06:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:50.514+0000 7f1953909540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-22T18:06:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:50.514+0000 7f1953909540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-22T18:06:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:50.514+0000 7f1953909540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-22T18:06:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:50.515+0000 7f1953909540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-06-22T18:06:51.039 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:51.035+0000 7f1953909540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:51.801 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:51.797+0000 7f1953909540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:52.564 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:52.560+0000 7f1953909540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:53.361 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-06-22T18:06:53.430 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:53.514 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:53.510+0000 7fa62ab26540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-06-22T18:06:53.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:53.510+0000 7fa62ab26540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-06-22T18:06:53.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:53.510+0000 7fa62ab26540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-22T18:06:53.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:53.510+0000 7fa62ab26540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-22T18:06:53.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:53.510+0000 7fa62ab26540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-22T18:06:53.516 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:53.511+0000 7fa62ab26540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-06-22T18:06:54.036 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:54.032+0000 7fa62ab26540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:54.717 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T18:06:54.719 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T18:06:54.806 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:54.802+0000 7fa62ab26540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:55.569 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:06:55.565+0000 7fa62ab26540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-22T18:06:56.360 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-06-22T18:06:56.429 INFO:tasks.ceph:Reading keys from all nodes... 2023-06-22T18:06:56.429 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:56.429 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-06-22T18:06:56.458 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:56.459 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-06-22T18:06:56.525 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:56.525 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-06-22T18:06:56.591 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:56.592 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-06-22T18:06:56.656 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:56.657 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-06-22T18:06:56.721 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:56.721 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-06-22T18:06:56.786 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:56.786 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-06-22T18:06:56.850 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:56.851 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-06-22T18:06:56.879 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:56.880 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-06-22T18:06:56.948 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:56.948 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-06-22T18:06:57.014 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:57.014 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-06-22T18:06:57.079 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:57.079 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-06-22T18:06:57.145 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:57.145 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-06-22T18:06:57.210 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:57.211 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-06-22T18:06:57.277 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:57.277 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-06-22T18:06:57.294 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:57.294 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-06-22T18:06:57.349 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:57.350 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-06-22T18:06:57.364 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:57.365 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-06-22T18:06:57.419 INFO:tasks.ceph:Adding keys to all mons... 2023-06-22T18:06:57.420 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-22T18:06:57.422 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-22T18:06:57.462 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2023-06-22T18:06:57.463 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCjjZRk07a2FxAAG3bM5QJxXmUCtThPlbE1Gg== 2023-06-22T18:06:57.463 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.a] 2023-06-22T18:06:57.463 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCjjZRk0HVbHhAAB+Fq/lR1S3xgEHI+wi7Amg== 2023-06-22T18:06:57.463 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.c] 2023-06-22T18:06:57.463 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCjjZRktV0wJRAATP4HtD+8fyj3cSp/p2xpsw== 2023-06-22T18:06:57.464 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2023-06-22T18:06:57.464 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCljZRkJBB8FxAAKoUF3Xy4YcQPtFv3UqGCgA== 2023-06-22T18:06:57.464 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2023-06-22T18:06:57.464 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCojZRkZP+jFxAAKpQ4kdRbkPzYWxY9x8gCmA== 2023-06-22T18:06:57.464 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2023-06-22T18:06:57.464 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCrjZRkt8izGBAAdvPK1mbewrCpzInHXbKKAw== 2023-06-22T18:06:57.464 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2023-06-22T18:06:57.465 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCujZRkKufZGRAA6ifZ7VYKYYwSUfEy89tt5w== 2023-06-22T18:06:57.465 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2023-06-22T18:06:57.465 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCjjZRkizReGxAAJ3e1e2I0g2TKmV7KuRW5DA== 2023-06-22T18:06:57.465 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.b] 2023-06-22T18:06:57.465 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCjjZRk6OjyKRAAyyiR41LHT1xA+aAaDWYs/Q== 2023-06-22T18:06:57.465 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.d] 2023-06-22T18:06:57.465 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCjjZRk7mfRMBAAV8ExxNqTNmGdsRWNrcWATw== 2023-06-22T18:06:57.466 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2023-06-22T18:06:57.466 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC0jZRkTB+xHRAA1TPybTCqL4vbD0/cSIds1Q== 2023-06-22T18:06:57.466 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2023-06-22T18:06:57.466 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC3jZRkcwt8HRAA5yR8B7R9Ym9N41COAR2Lxw== 2023-06-22T18:06:57.466 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2023-06-22T18:06:57.466 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC6jZRkS0zCHhAAWVmUStf1UVbkX8yD70kcnw== 2023-06-22T18:06:57.467 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2023-06-22T18:06:57.467 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC9jZRk5F2EHhAAXx4f4NwLcLqSU6aIIeIiHw== 2023-06-22T18:06:57.467 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2023-06-22T18:06:57.467 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCjjZRkxg62NBAAdww8+svTSJs7LP68GqajDg== 2023-06-22T18:06:57.467 INFO:teuthology.orchestra.run.smithi110.stdout:[client.2] 2023-06-22T18:06:57.467 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCjjZRkj5IjOhAAh1LdhnJXXBY0m6wxqNx5PA== 2023-06-22T18:06:57.467 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2023-06-22T18:06:57.468 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCkjZRktv6UARAAgso9mNbV5sQBgc9m7BMUqg== 2023-06-22T18:06:57.468 INFO:teuthology.orchestra.run.smithi110.stdout:[client.3] 2023-06-22T18:06:57.468 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCkjZRkK3YEBxAAT/r5XvCEM0+wYXlZRYOEkQ== 2023-06-22T18:06:57.482 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.y] 2023-06-22T18:06:57.483 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCjjZRk07a2FxAAG3bM5QJxXmUCtThPlbE1Gg== 2023-06-22T18:06:57.483 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2023-06-22T18:06:57.483 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCjjZRk0HVbHhAAB+Fq/lR1S3xgEHI+wi7Amg== 2023-06-22T18:06:57.484 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2023-06-22T18:06:57.484 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCjjZRktV0wJRAATP4HtD+8fyj3cSp/p2xpsw== 2023-06-22T18:06:57.484 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2023-06-22T18:06:57.484 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCljZRkJBB8FxAAKoUF3Xy4YcQPtFv3UqGCgA== 2023-06-22T18:06:57.484 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2023-06-22T18:06:57.484 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCojZRkZP+jFxAAKpQ4kdRbkPzYWxY9x8gCmA== 2023-06-22T18:06:57.485 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2023-06-22T18:06:57.485 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCrjZRkt8izGBAAdvPK1mbewrCpzInHXbKKAw== 2023-06-22T18:06:57.485 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.3] 2023-06-22T18:06:57.485 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCujZRkKufZGRAA6ifZ7VYKYYwSUfEy89tt5w== 2023-06-22T18:06:57.485 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2023-06-22T18:06:57.485 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCjjZRkizReGxAAJ3e1e2I0g2TKmV7KuRW5DA== 2023-06-22T18:06:57.485 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2023-06-22T18:06:57.485 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCjjZRk6OjyKRAAyyiR41LHT1xA+aAaDWYs/Q== 2023-06-22T18:06:57.486 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2023-06-22T18:06:57.486 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCjjZRk7mfRMBAAV8ExxNqTNmGdsRWNrcWATw== 2023-06-22T18:06:57.486 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.4] 2023-06-22T18:06:57.486 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQC0jZRkTB+xHRAA1TPybTCqL4vbD0/cSIds1Q== 2023-06-22T18:06:57.486 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.5] 2023-06-22T18:06:57.486 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQC3jZRkcwt8HRAA5yR8B7R9Ym9N41COAR2Lxw== 2023-06-22T18:06:57.487 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.6] 2023-06-22T18:06:57.487 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQC6jZRkS0zCHhAAWVmUStf1UVbkX8yD70kcnw== 2023-06-22T18:06:57.487 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.7] 2023-06-22T18:06:57.487 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQC9jZRk5F2EHhAAXx4f4NwLcLqSU6aIIeIiHw== 2023-06-22T18:06:57.487 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2023-06-22T18:06:57.487 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCjjZRkxg62NBAAdww8+svTSJs7LP68GqajDg== 2023-06-22T18:06:57.487 INFO:teuthology.orchestra.run.smithi116.stdout:[client.2] 2023-06-22T18:06:57.487 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCjjZRkj5IjOhAAh1LdhnJXXBY0m6wxqNx5PA== 2023-06-22T18:06:57.488 INFO:teuthology.orchestra.run.smithi116.stdout:[client.1] 2023-06-22T18:06:57.488 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCkjZRktv6UARAAgso9mNbV5sQBgc9m7BMUqg== 2023-06-22T18:06:57.488 INFO:teuthology.orchestra.run.smithi116.stdout:[client.3] 2023-06-22T18:06:57.488 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQCkjZRkK3YEBxAAT/r5XvCEM0+wYXlZRYOEkQ== 2023-06-22T18:06:57.489 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-22T18:06:57.511 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-22T18:06:57.575 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T18:06:57.578 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T18:06:57.628 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T18:06:57.631 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T18:06:57.681 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:57.686 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:57.735 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:57.776 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:57.826 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:57.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:57.881 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:57.885 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:57.934 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-22T18:06:57.938 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-22T18:06:57.987 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T18:06:57.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T18:06:58.040 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T18:06:58.042 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-22T18:06:58.092 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:58.094 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:58.141 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:58.145 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:58.193 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:58.196 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:58.243 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:58.245 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-22T18:06:58.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-22T18:06:58.295 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-22T18:06:58.345 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-22T18:06:58.347 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-22T18:06:58.397 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-22T18:06:58.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-22T18:06:58.449 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-22T18:06:58.451 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-22T18:06:58.501 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-06-22T18:06:58.501 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-06-22T18:06:58.525 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-22T18:06:58.616 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-06-22T18:06:58.640 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-06-22T18:06:58.664 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-22T18:06:58.755 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-06-22T18:06:58.778 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-06-22T18:06:58.843 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-22T18:06:58.933 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-06-22T18:06:58.957 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:58.960 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-22T18:06:59.012 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-06-22T18:06:59.013 INFO:tasks.ceph.mon.a:Restarting daemon 2023-06-22T18:06:59.013 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-06-22T18:06:59.015 INFO:tasks.ceph.mon.a:Started 2023-06-22T18:06:59.016 INFO:tasks.ceph.mon.b:Restarting daemon 2023-06-22T18:06:59.016 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-06-22T18:06:59.055 INFO:tasks.ceph.mon.b:Started 2023-06-22T18:06:59.055 INFO:tasks.ceph.mon.c:Restarting daemon 2023-06-22T18:06:59.056 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-06-22T18:06:59.057 INFO:tasks.ceph.mon.c:Started 2023-06-22T18:06:59.058 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-06-22T18:06:59.058 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-06-22T18:06:59.058 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-06-22T18:06:59.060 INFO:tasks.ceph.mgr.y:Started 2023-06-22T18:06:59.060 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-06-22T18:06:59.061 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-06-22T18:06:59.062 INFO:tasks.ceph.mgr.x:Started 2023-06-22T18:06:59.063 INFO:tasks.ceph:Setting crush tunables to default 2023-06-22T18:06:59.063 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd crush tunables default 2023-06-22T18:06:59.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.174+0000 7efeeb311700 1 Processor -- start 2023-06-22T18:06:59.178 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.175+0000 7efeeb311700 1 -- start start 2023-06-22T18:06:59.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.175+0000 7efeeb311700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efee409d7e0 0x7efee409dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:06:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.175+0000 7efeeb311700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 0x7efee40cca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:06:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.175+0000 7efeeb311700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee40b3220 con 0x7efee40ccf80 2023-06-22T18:06:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.175+0000 7efeeb311700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee40b37e0 con 0x7efee409d7e0 2023-06-22T18:06:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.175+0000 7efeeb311700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee40cd500 con 0x7efee40b3c70 2023-06-22T18:06:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.175+0000 7efee90ad700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efee409d7e0 0x7efee409dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:06:59.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.175+0000 7efee90ad700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efee409d7e0 0x7efee409dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33648/0 (socket says 172.21.15.110:33648) 2023-06-22T18:06:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.176+0000 7efee90ad700 1 -- 172.21.15.110:0/1041615300 learned_addr learned my addr 172.21.15.110:0/1041615300 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:06:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.176+0000 7efee90ad700 1 -- 172.21.15.110:0/1041615300 >> v2:172.21.15.110:3300/0 conn(0x7efee409d7e0 msgr2=0x7efee409dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-22T18:06:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.176+0000 7efee90ad700 1 -- 172.21.15.110:0/1041615300 >> v2:172.21.15.110:3300/0 conn(0x7efee409d7e0 msgr2=0x7efee409dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T18:06:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.176+0000 7efee90ad700 1 --2- 172.21.15.110:0/1041615300 >> v2:172.21.15.110:3300/0 conn(0x7efee409d7e0 0x7efee409dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-22T18:06:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.176+0000 7efee90ad700 1 --2- 172.21.15.110:0/1041615300 >> v2:172.21.15.110:3300/0 conn(0x7efee409d7e0 0x7efee409dbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T18:06:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.176+0000 7efee88ac700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 0x7efee40cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:06:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.176+0000 7efee88ac700 1 -- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 msgr2=0x7efee40cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-06-22T18:06:59.181 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.176+0000 7efee88ac700 1 -- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 msgr2=0x7efee40cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-22T18:06:59.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.176+0000 7efee88ac700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 0x7efee40cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-22T18:06:59.182 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.176+0000 7efee88ac700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 0x7efee40cca40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-22T18:06:59.355 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823293084 0 0) 0x7efee40b3220 con 0x7efee40ccf80 2023-06-22T18:06:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed0003570 con 0x7efee40ccf80 2023-06-22T18:06:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 <== mon.1 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1217141620 0 0) 0x7efed0003570 con 0x7efee40ccf80 2023-06-22T18:06:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 msgr2=0x7efee40cca40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T18:06:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 0x7efee40cca40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:06:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 >> v2:172.21.15.110:3300/0 conn(0x7efee409d7e0 msgr2=0x7efee409dbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-22T18:06:59.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 --2- 172.21.15.110:0/1041615300 >> v2:172.21.15.110:3300/0 conn(0x7efee409d7e0 0x7efee409dbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:06:59.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee40cd7e0 con 0x7efee40ccf80 2023-06-22T18:06:59.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 <== mon.1 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 519278953 0 0) 0x7efedc0017f0 con 0x7efee40ccf80 2023-06-22T18:06:59.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 >> v1:172.21.15.116:6789/0 conn(0x7efee40ccf80 legacy=0x7efee40cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:06:59.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 0x7efed0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:06:59.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 0x7efed0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:06:59.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed0004d50 0x7efed00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:06:59.357 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee40b37e0 con 0x7efed0003a60 2023-06-22T18:06:59.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee40cd500 con 0x7efed0004d50 2023-06-22T18:06:59.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed000bb90 con 0x7efee40b3c70 2023-06-22T18:06:59.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efee90ad700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 0x7efed0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:06:59.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.351+0000 7efee88ac700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 0x7efed0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:06:59.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.352+0000 7efee98ae700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed0004d50 0x7efed00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:06:59.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.352+0000 7efee90ad700 1 -- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 msgr2=0x7efed0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:06:59.358 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.352+0000 7efee90ad700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40b3c70 0x7efed0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:06:59.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.352+0000 7efee90ad700 1 -- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed0004d50 msgr2=0x7efed00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:06:59.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.352+0000 7efee90ad700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed0004d50 0x7efed00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:06:59.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.352+0000 7efee90ad700 1 -- 172.21.15.110:0/1041615300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0003570 con 0x7efed0003a60 2023-06-22T18:06:59.359 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.352+0000 7efee90ad700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 0x7efed0003e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efee0009060 tx=0x7efee0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb23d1e54cd09ed server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:06:59.361 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.353+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee000c070 con 0x7efed0003a60 2023-06-22T18:06:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.353+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee000e5e0 con 0x7efed0003a60 2023-06-22T18:06:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.353+0000 7efedbfff700 1 -- 172.21.15.110:0/1041615300 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee000e770 con 0x7efed0003a60 2023-06-22T18:06:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.354+0000 7efeeb311700 1 -- 172.21.15.110:0/1041615300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 msgr2=0x7efed0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:06:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.354+0000 7efeeb311700 1 --2- 172.21.15.110:0/1041615300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 0x7efed0003e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efee0009060 tx=0x7efee0008040 comp rx=0 tx=0).stop 2023-06-22T18:06:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.354+0000 7efee98ae700 1 -- 172.21.15.110:0/1041615300 reap_dead start 2023-06-22T18:06:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.354+0000 7efeeb311700 1 -- 172.21.15.110:0/1041615300 shutdown_connections 2023-06-22T18:06:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.354+0000 7efeeb311700 1 -- 172.21.15.110:0/1041615300 >> 172.21.15.110:0/1041615300 conn(0x7efee4067340 msgr2=0x7efee4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:06:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.355+0000 7efeeb311700 1 -- 172.21.15.110:0/1041615300 shutdown_connections 2023-06-22T18:06:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.355+0000 7efeeb311700 1 -- 172.21.15.110:0/1041615300 wait complete. 2023-06-22T18:06:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.356+0000 7efeeb311700 1 Processor -- start 2023-06-22T18:06:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.356+0000 7efeeb311700 1 -- start start 2023-06-22T18:06:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efeeb311700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 0x7efee4152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:06:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efeeb311700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40c7620 0x7efee41591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:06:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efeeb311700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4157c70 0x7efee40cf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:06:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efeeb311700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee40c7a90 con 0x7efed0003a60 2023-06-22T18:06:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efeeb311700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee415a750 con 0x7efee4157c70 2023-06-22T18:06:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efeeb311700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee41583a0 con 0x7efee40c7620 2023-06-22T18:06:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee90ad700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 0x7efee4152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:06:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee90ad700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 0x7efee4152480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33686/0 (socket says 172.21.15.110:33686) 2023-06-22T18:06:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee90ad700 1 -- 172.21.15.110:0/2357673353 learned_addr learned my addr 172.21.15.110:0/2357673353 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:06:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee98ae700 1 --2- 172.21.15.110:0/2357673353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4157c70 0x7efee40cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:06:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee88ac700 1 --2- 172.21.15.110:0/2357673353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40c7620 0x7efee41591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:06:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee90ad700 1 -- 172.21.15.110:0/2357673353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40c7620 msgr2=0x7efee41591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:06:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee90ad700 1 --2- 172.21.15.110:0/2357673353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40c7620 0x7efee41591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:06:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee90ad700 1 -- 172.21.15.110:0/2357673353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4157c70 msgr2=0x7efee40cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:06:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee90ad700 1 --2- 172.21.15.110:0/2357673353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4157c70 0x7efee40cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:06:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee90ad700 1 -- 172.21.15.110:0/2357673353 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee41580e0 con 0x7efed0003a60 2023-06-22T18:06:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee98ae700 1 --2- 172.21.15.110:0/2357673353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4157c70 0x7efee40cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T18:06:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.357+0000 7efee90ad700 1 --2- 172.21.15.110:0/2357673353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 0x7efee4152480 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efee0013770 tx=0x7efee001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:06:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.358+0000 7efed97fa700 1 -- 172.21.15.110:0/2357673353 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0013c10 con 0x7efed0003a60 2023-06-22T18:06:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.358+0000 7efed97fa700 1 -- 172.21.15.110:0/2357673353 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee0013d70 con 0x7efed0003a60 2023-06-22T18:06:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.358+0000 7efeeb311700 1 -- 172.21.15.110:0/2357673353 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee4159e20 con 0x7efed0003a60 2023-06-22T18:06:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.358+0000 7efed97fa700 1 -- 172.21.15.110:0/2357673353 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee001d6b0 con 0x7efed0003a60 2023-06-22T18:06:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.358+0000 7efeeb311700 1 -- 172.21.15.110:0/2357673353 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee415a3e0 con 0x7efed0003a60 2023-06-22T18:06:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.358+0000 7efed97fa700 1 -- 172.21.15.110:0/2357673353 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7efee001dae0 con 0x7efed0003a60 2023-06-22T18:06:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.358+0000 7efed97fa700 1 -- 172.21.15.110:0/2357673353 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7efee002f790 con 0x7efed0003a60 2023-06-22T18:06:59.367 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.359+0000 7efeeb311700 1 -- 172.21.15.110:0/2357673353 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee415a570 con 0x7efed0003a60 2023-06-22T18:06:59.367 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.361+0000 7efed97fa700 1 -- 172.21.15.110:0/2357673353 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7efee002a070 con 0x7efed0003a60 2023-06-22T18:06:59.460 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.457+0000 7efeeb311700 1 -- 172.21.15.110:0/2357673353 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7efee404c1b0 con 0x7efed0003a60 2023-06-22T18:06:59.465 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.462+0000 7efed97fa700 1 -- 172.21.15.110:0/2357673353 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7efee0017030 con 0x7efed0003a60 2023-06-22T18:06:59.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.463+0000 7efeeb311700 1 -- 172.21.15.110:0/2357673353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 msgr2=0x7efee4152480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:06:59.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.463+0000 7efeeb311700 1 --2- 172.21.15.110:0/2357673353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 0x7efee4152480 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efee0013770 tx=0x7efee001a000 comp rx=0 tx=0).stop 2023-06-22T18:06:59.466 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.463+0000 7efeeb311700 1 -- 172.21.15.110:0/2357673353 shutdown_connections 2023-06-22T18:06:59.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.463+0000 7efeeb311700 1 --2- 172.21.15.110:0/2357673353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee4157c70 0x7efee40cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:06:59.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.463+0000 7efeeb311700 1 --2- 172.21.15.110:0/2357673353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efed0003a60 0x7efee4152480 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:06:59.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.463+0000 7efeeb311700 1 --2- 172.21.15.110:0/2357673353 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7efee40c7620 0x7efee41591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:06:59.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.463+0000 7efeeb311700 1 -- 172.21.15.110:0/2357673353 >> 172.21.15.110:0/2357673353 conn(0x7efee4067340 msgr2=0x7efee40d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:06:59.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.463+0000 7efeeb311700 1 -- 172.21.15.110:0/2357673353 shutdown_connections 2023-06-22T18:06:59.467 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:06:59.463+0000 7efeeb311700 1 -- 172.21.15.110:0/2357673353 wait complete. 2023-06-22T18:06:59.468 INFO:teuthology.orchestra.run.smithi110.stderr:adjusted tunables profile to default 2023-06-22T18:06:59.477 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-06-22T18:06:59.478 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:59.478 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-06-22T18:06:59.501 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:59.502 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-06-22T18:06:59.566 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:59.566 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-06-22T18:06:59.630 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-06-22T18:06:59.631 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-06-22T18:06:59.694 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:59.694 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-06-22T18:06:59.719 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:59.719 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-06-22T18:06:59.783 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:59.783 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-06-22T18:06:59.848 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2023-06-22T18:06:59.848 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-06-22T18:06:59.912 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 66c18197-0807-4f09-9e82-decc44f8bb6c 0 2023-06-22T18:07:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.062+0000 7f01631a0700 1 Processor -- start 2023-06-22T18:07:00.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.062+0000 7f01631a0700 1 -- start start 2023-06-22T18:07:00.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f01631a0700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f015c06a080 0x7f015c0ca890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f01631a0700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a450 0x7f015c15d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f01631a0700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c159a60 con 0x7f015c0cee50 2023-06-22T18:07:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f01631a0700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015c0afac0 con 0x7f015c06a080 2023-06-22T18:07:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f01631a0700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0cadd0 con 0x7f015c06a450 2023-06-22T18:07:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f016299f700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a450 0x7f015c15d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f016199d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f015c0cee50 0x7f015c15a1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:38712/0 (socket says 172.21.15.116:38712) 2023-06-22T18:07:00.068 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f016199d700 1 -- 172.21.15.116:0/1341762844 learned_addr learned my addr 172.21.15.116:0/1341762844 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-06-22T18:07:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f016219e700 1 --2- 172.21.15.116:0/1341762844 >> v2:172.21.15.110:3300/0 conn(0x7f015c06a080 0x7f015c0ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f016299f700 1 -- 172.21.15.116:0/1341762844 >> v2:172.21.15.110:3300/0 conn(0x7f015c06a080 msgr2=0x7f015c0ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f016299f700 1 --2- 172.21.15.116:0/1341762844 >> v2:172.21.15.110:3300/0 conn(0x7f015c06a080 0x7f015c0ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f016299f700 1 -- 172.21.15.116:0/1341762844 >> v1:172.21.15.116:6789/0 conn(0x7f015c0cee50 legacy=0x7f015c15a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f016299f700 1 -- 172.21.15.116:0/1341762844 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c0cb1d0 con 0x7f015c06a450 2023-06-22T18:07:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f016119c700 1 -- 172.21.15.116:0/1341762844 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113138283 0 0) 0x7f015c159a60 con 0x7f015c0cee50 2023-06-22T18:07:00.069 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f016219e700 1 --2- 172.21.15.116:0/1341762844 >> v2:172.21.15.110:3300/0 conn(0x7f015c06a080 0x7f015c0ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T18:07:00.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.063+0000 7f016299f700 1 --2- 172.21.15.116:0/1341762844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a450 0x7f015c15d060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0154009e60 tx=0x7f0154009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8da4423c7f6277d2 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:00.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.064+0000 7f016119c700 1 -- 172.21.15.116:0/1341762844 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015400e070 con 0x7f015c06a450 2023-06-22T18:07:00.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.064+0000 7f016119c700 1 -- 172.21.15.116:0/1341762844 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015400bc30 con 0x7f015c06a450 2023-06-22T18:07:00.070 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.064+0000 7f016119c700 1 -- 172.21.15.116:0/1341762844 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154012430 con 0x7f015c06a450 2023-06-22T18:07:00.072 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.064+0000 7f01631a0700 1 -- 172.21.15.116:0/1341762844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a450 msgr2=0x7f015c15d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.064+0000 7f01631a0700 1 --2- 172.21.15.116:0/1341762844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a450 0x7f015c15d060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0154009e60 tx=0x7f0154009780 comp rx=0 tx=0).stop 2023-06-22T18:07:00.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.064+0000 7f01631a0700 1 -- 172.21.15.116:0/1341762844 shutdown_connections 2023-06-22T18:07:00.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.064+0000 7f01631a0700 1 --2- 172.21.15.116:0/1341762844 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a450 0x7f015c15d060 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.064+0000 7f01631a0700 1 --2- 172.21.15.116:0/1341762844 >> v2:172.21.15.110:3300/0 conn(0x7f015c06a080 0x7f015c0ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.064+0000 7f01631a0700 1 -- 172.21.15.116:0/1341762844 >> 172.21.15.116:0/1341762844 conn(0x7f015c0d0250 msgr2=0x7f015c0d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:00.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.064+0000 7f01631a0700 1 -- 172.21.15.116:0/1341762844 shutdown_connections 2023-06-22T18:07:00.073 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.064+0000 7f01631a0700 1 -- 172.21.15.116:0/1341762844 wait complete. 2023-06-22T18:07:00.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.065+0000 7f01631a0700 1 Processor -- start 2023-06-22T18:07:00.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.065+0000 7f01631a0700 1 -- start start 2023-06-22T18:07:00.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f01631a0700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a080 0x7f015c1576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f01631a0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06a450 0x7f015c0c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f01631a0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0cee50 0x7f015c1587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f01631a0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0c8740 con 0x7f015c06a450 2023-06-22T18:07:00.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f01631a0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0c7b30 con 0x7f015c0cee50 2023-06-22T18:07:00.074 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f01631a0700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0c7e10 con 0x7f015c06a080 2023-06-22T18:07:00.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f016219e700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a080 0x7f015c1576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f016299f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0cee50 0x7f015c1587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f016219e700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a080 0x7f015c1576f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48066/0 (socket says 172.21.15.116:48066) 2023-06-22T18:07:00.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f016219e700 1 -- 172.21.15.116:0/934708168 learned_addr learned my addr 172.21.15.116:0/934708168 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-06-22T18:07:00.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f016199d700 1 --2- 172.21.15.116:0/934708168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06a450 0x7f015c0c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f016299f700 1 -- 172.21.15.116:0/934708168 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a080 msgr2=0x7f015c1576f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f016299f700 1 --2- 172.21.15.116:0/934708168 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a080 0x7f015c1576f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.075 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f016299f700 1 -- 172.21.15.116:0/934708168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06a450 msgr2=0x7f015c0c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f016299f700 1 --2- 172.21.15.116:0/934708168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06a450 0x7f015c0c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.066+0000 7f016299f700 1 -- 172.21.15.116:0/934708168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154009a20 con 0x7f015c0cee50 2023-06-22T18:07:00.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.067+0000 7f016219e700 1 --2- 172.21.15.116:0/934708168 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a080 0x7f015c1576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:00.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.067+0000 7f016299f700 1 --2- 172.21.15.116:0/934708168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0cee50 0x7f015c1587e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0154016000 tx=0x7f0154019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:00.076 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.067+0000 7f01527fc700 1 -- 172.21.15.116:0/934708168 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154027470 con 0x7f015c0cee50 2023-06-22T18:07:00.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.067+0000 7f01527fc700 1 -- 172.21.15.116:0/934708168 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01540275f0 con 0x7f015c0cee50 2023-06-22T18:07:00.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.067+0000 7f01527fc700 1 -- 172.21.15.116:0/934708168 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01540279b0 con 0x7f015c0cee50 2023-06-22T18:07:00.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.067+0000 7f01631a0700 1 -- 172.21.15.116:0/934708168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c0c80f0 con 0x7f015c0cee50 2023-06-22T18:07:00.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.067+0000 7f01631a0700 1 -- 172.21.15.116:0/934708168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c0c7840 con 0x7f015c0cee50 2023-06-22T18:07:00.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.067+0000 7f01527fc700 1 -- 172.21.15.116:0/934708168 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0154024070 con 0x7f015c0cee50 2023-06-22T18:07:00.077 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.067+0000 7f01527fc700 1 -- 172.21.15.116:0/934708168 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f015401d630 con 0x7f015c0cee50 2023-06-22T18:07:00.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.068+0000 7f01631a0700 1 -- 172.21.15.116:0/934708168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c070e80 con 0x7f015c0cee50 2023-06-22T18:07:00.078 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.070+0000 7f01527fc700 1 -- 172.21.15.116:0/934708168 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f01540476e0 con 0x7f015c0cee50 2023-06-22T18:07:00.155 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.152+0000 7f01631a0700 1 -- 172.21.15.116:0/934708168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "66c18197-0807-4f09-9e82-decc44f8bb6c", "id": 0} v 0) v1 -- 0x7f015c15e1e0 con 0x7f015c0cee50 2023-06-22T18:07:00.159 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.155+0000 7f01527fc700 1 -- 172.21.15.116:0/934708168 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "66c18197-0807-4f09-9e82-decc44f8bb6c", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0154036030 con 0x7f015c0cee50 2023-06-22T18:07:00.159 INFO:teuthology.orchestra.run.smithi116.stdout:0 2023-06-22T18:07:00.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.156+0000 7f01631a0700 1 -- 172.21.15.116:0/934708168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0cee50 msgr2=0x7f015c1587e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.156+0000 7f01631a0700 1 --2- 172.21.15.116:0/934708168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0cee50 0x7f015c1587e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0154016000 tx=0x7f0154019000 comp rx=0 tx=0).stop 2023-06-22T18:07:00.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.156+0000 7f01631a0700 1 -- 172.21.15.116:0/934708168 shutdown_connections 2023-06-22T18:07:00.160 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.156+0000 7f01631a0700 1 --2- 172.21.15.116:0/934708168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0cee50 0x7f015c1587e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.156+0000 7f01631a0700 1 --2- 172.21.15.116:0/934708168 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f015c06a080 0x7f015c1576f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.156+0000 7f01631a0700 1 --2- 172.21.15.116:0/934708168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06a450 0x7f015c0c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.156+0000 7f01631a0700 1 -- 172.21.15.116:0/934708168 >> 172.21.15.116:0/934708168 conn(0x7f015c0d0250 msgr2=0x7f015c0aefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:00.161 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.156+0000 7f01631a0700 1 -- 172.21.15.116:0/934708168 shutdown_connections 2023-06-22T18:07:00.162 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.156+0000 7f01631a0700 1 -- 172.21.15.116:0/934708168 wait complete. 2023-06-22T18:07:00.171 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new e2e2e368-b287-4ae9-a502-f2d0276de124 1 2023-06-22T18:07:00.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.282+0000 7f85d24e0700 1 Processor -- start 2023-06-22T18:07:00.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.282+0000 7f85d24e0700 1 -- start start 2023-06-22T18:07:00.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d24e0700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f85cc142a90 0x7f85cc0b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d24e0700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc0b3cb0 0x7f85cc069420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d24e0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f85cc069960 0x7f85cc159890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d24e0700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85cc0b16e0 con 0x7f85cc142a90 2023-06-22T18:07:00.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d24e0700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85cc068660 con 0x7f85cc069960 2023-06-22T18:07:00.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d24e0700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc068940 con 0x7f85cc0b3cb0 2023-06-22T18:07:00.289 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d1cdf700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f85cc069960 0x7f85cc159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d0cdd700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc0b3cb0 0x7f85cc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d1cdf700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f85cc069960 0x7f85cc159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38786/0 (socket says 172.21.15.116:38786) 2023-06-22T18:07:00.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d0cdd700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc0b3cb0 0x7f85cc069420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48068/0 (socket says 172.21.15.116:48068) 2023-06-22T18:07:00.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d1cdf700 1 -- 172.21.15.116:0/3641774880 learned_addr learned my addr 172.21.15.116:0/3641774880 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-06-22T18:07:00.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d14de700 1 --2- 172.21.15.116:0/3641774880 >> v2:172.21.15.110:3300/0 conn(0x7f85cc142a90 0x7f85cc0b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.290 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d0cdd700 1 -- 172.21.15.116:0/3641774880 >> v2:172.21.15.116:3300/0 conn(0x7f85cc069960 msgr2=0x7f85cc159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:00.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d0cdd700 1 --2- 172.21.15.116:0/3641774880 >> v2:172.21.15.116:3300/0 conn(0x7f85cc069960 0x7f85cc159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d0cdd700 1 -- 172.21.15.116:0/3641774880 >> v2:172.21.15.110:3300/0 conn(0x7f85cc142a90 msgr2=0x7f85cc0b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:00.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d0cdd700 1 --2- 172.21.15.116:0/3641774880 >> v2:172.21.15.110:3300/0 conn(0x7f85cc142a90 0x7f85cc0b0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d0cdd700 1 -- 172.21.15.116:0/3641774880 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85cc068c20 con 0x7f85cc0b3cb0 2023-06-22T18:07:00.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.283+0000 7f85d1cdf700 1 --2- 172.21.15.116:0/3641774880 >> v2:172.21.15.116:3300/0 conn(0x7f85cc069960 0x7f85cc159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:00.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85d0cdd700 1 --2- 172.21.15.116:0/3641774880 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc0b3cb0 0x7f85cc069420 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f85cc06a970 tx=0x7f85c80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd1e14e88f691c20 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:00.291 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85c3fff700 1 -- 172.21.15.116:0/3641774880 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8011630 con 0x7f85cc0b3cb0 2023-06-22T18:07:00.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85c3fff700 1 -- 172.21.15.116:0/3641774880 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c80117b0 con 0x7f85cc0b3cb0 2023-06-22T18:07:00.292 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85c3fff700 1 -- 172.21.15.116:0/3641774880 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8011b70 con 0x7f85cc0b3cb0 2023-06-22T18:07:00.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85d24e0700 1 -- 172.21.15.116:0/3641774880 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc0b3cb0 msgr2=0x7f85cc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.296 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85d24e0700 1 --2- 172.21.15.116:0/3641774880 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc0b3cb0 0x7f85cc069420 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f85cc06a970 tx=0x7f85c80091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85d24e0700 1 -- 172.21.15.116:0/3641774880 shutdown_connections 2023-06-22T18:07:00.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85d24e0700 1 --2- 172.21.15.116:0/3641774880 >> v2:172.21.15.110:3300/0 conn(0x7f85cc142a90 0x7f85cc0b0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85d24e0700 1 --2- 172.21.15.116:0/3641774880 >> v2:172.21.15.116:3300/0 conn(0x7f85cc069960 0x7f85cc159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.297 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85d24e0700 1 --2- 172.21.15.116:0/3641774880 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc0b3cb0 0x7f85cc069420 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85d24e0700 1 -- 172.21.15.116:0/3641774880 >> 172.21.15.116:0/3641774880 conn(0x7f85cc070220 msgr2=0x7f85cc070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:00.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85d24e0700 1 -- 172.21.15.116:0/3641774880 shutdown_connections 2023-06-22T18:07:00.298 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.284+0000 7f85d24e0700 1 -- 172.21.15.116:0/3641774880 wait complete. 2023-06-22T18:07:00.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.286+0000 7f85d24e0700 1 Processor -- start 2023-06-22T18:07:00.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.286+0000 7f85d24e0700 1 -- start start 2023-06-22T18:07:00.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.286+0000 7f85d24e0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc069960 0x7f85cc0c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.286+0000 7f85d24e0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85cc0b3cb0 0x7f85cc1698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.286+0000 7f85d24e0700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc142a90 0x7f85cc16c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.299 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.286+0000 7f85d24e0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc16a920 con 0x7f85cc069960 2023-06-22T18:07:00.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.286+0000 7f85d24e0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc0a5470 con 0x7f85cc0b3cb0 2023-06-22T18:07:00.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.286+0000 7f85d24e0700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc0c9cf0 con 0x7f85cc142a90 2023-06-22T18:07:00.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d0cdd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85cc0b3cb0 0x7f85cc1698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.300 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d1cdf700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc142a90 0x7f85cc16c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d1cdf700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc142a90 0x7f85cc16c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48070/0 (socket says 172.21.15.116:48070) 2023-06-22T18:07:00.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d0cdd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85cc0b3cb0 0x7f85cc1698d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38800/0 (socket says 172.21.15.116:38800) 2023-06-22T18:07:00.301 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d1cdf700 1 -- 172.21.15.116:0/2689766061 learned_addr learned my addr 172.21.15.116:0/2689766061 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-06-22T18:07:00.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d14de700 1 --2- 172.21.15.116:0/2689766061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc069960 0x7f85cc0c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d1cdf700 1 -- 172.21.15.116:0/2689766061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85cc0b3cb0 msgr2=0x7f85cc1698d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.302 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d1cdf700 1 --2- 172.21.15.116:0/2689766061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85cc0b3cb0 0x7f85cc1698d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d1cdf700 1 -- 172.21.15.116:0/2689766061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc069960 msgr2=0x7f85cc0c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d1cdf700 1 --2- 172.21.15.116:0/2689766061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc069960 0x7f85cc0c97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d1cdf700 1 -- 172.21.15.116:0/2689766061 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c4009ae0 con 0x7f85cc142a90 2023-06-22T18:07:00.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d0cdd700 1 --2- 172.21.15.116:0/2689766061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85cc0b3cb0 0x7f85cc1698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:00.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d1cdf700 1 --2- 172.21.15.116:0/2689766061 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc142a90 0x7f85cc16c770 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f85c4009fd0 tx=0x7f85c400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:00.303 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85c17fa700 1 -- 172.21.15.116:0/2689766061 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c4018070 con 0x7f85cc142a90 2023-06-22T18:07:00.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85c17fa700 1 -- 172.21.15.116:0/2689766061 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c400f040 con 0x7f85cc142a90 2023-06-22T18:07:00.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d24e0700 1 -- 172.21.15.116:0/2689766061 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c8009420 con 0x7f85cc142a90 2023-06-22T18:07:00.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85c17fa700 1 -- 172.21.15.116:0/2689766061 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c400cbd0 con 0x7f85cc142a90 2023-06-22T18:07:00.304 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85d24e0700 1 -- 172.21.15.116:0/2689766061 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85cc16a080 con 0x7f85cc142a90 2023-06-22T18:07:00.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85c17fa700 1 -- 172.21.15.116:0/2689766061 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f85c4003020 con 0x7f85cc142a90 2023-06-22T18:07:00.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.287+0000 7f85c17fa700 1 -- 172.21.15.116:0/2689766061 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f85c401d410 con 0x7f85cc142a90 2023-06-22T18:07:00.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.288+0000 7f85d24e0700 1 -- 172.21.15.116:0/2689766061 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85cc0710c0 con 0x7f85cc142a90 2023-06-22T18:07:00.305 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.290+0000 7f85c17fa700 1 -- 172.21.15.116:0/2689766061 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f85c4014480 con 0x7f85cc142a90 2023-06-22T18:07:00.379 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.375+0000 7f85d24e0700 1 -- 172.21.15.116:0/2689766061 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e2e2e368-b287-4ae9-a502-f2d0276de124", "id": 1} v 0) v1 -- 0x7f85cc0ad6b0 con 0x7f85cc142a90 2023-06-22T18:07:00.382 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.378+0000 7f85c17fa700 1 -- 172.21.15.116:0/2689766061 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e2e2e368-b287-4ae9-a502-f2d0276de124", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f85c402c0b0 con 0x7f85cc142a90 2023-06-22T18:07:00.382 INFO:teuthology.orchestra.run.smithi116.stdout:1 2023-06-22T18:07:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.379+0000 7f85d24e0700 1 -- 172.21.15.116:0/2689766061 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc142a90 msgr2=0x7f85cc16c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.379+0000 7f85d24e0700 1 --2- 172.21.15.116:0/2689766061 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc142a90 0x7f85cc16c770 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f85c4009fd0 tx=0x7f85c400b040 comp rx=0 tx=0).stop 2023-06-22T18:07:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.379+0000 7f85d24e0700 1 -- 172.21.15.116:0/2689766061 shutdown_connections 2023-06-22T18:07:00.383 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.379+0000 7f85d24e0700 1 --2- 172.21.15.116:0/2689766061 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f85cc142a90 0x7f85cc16c770 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.379+0000 7f85d24e0700 1 --2- 172.21.15.116:0/2689766061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc069960 0x7f85cc0c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.379+0000 7f85d24e0700 1 --2- 172.21.15.116:0/2689766061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85cc0b3cb0 0x7f85cc1698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.379+0000 7f85d24e0700 1 -- 172.21.15.116:0/2689766061 >> 172.21.15.116:0/2689766061 conn(0x7f85cc070220 msgr2=0x7f85cc141be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.379+0000 7f85d24e0700 1 -- 172.21.15.116:0/2689766061 shutdown_connections 2023-06-22T18:07:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.379+0000 7f85d24e0700 1 -- 172.21.15.116:0/2689766061 wait complete. 2023-06-22T18:07:00.398 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 89608384-c8bd-4032-98e6-6dd77f29f4de 2 2023-06-22T18:07:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.510+0000 7ff23f1ae700 1 Processor -- start 2023-06-22T18:07:00.515 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.511+0000 7ff23f1ae700 1 -- start start 2023-06-22T18:07:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.511+0000 7ff23f1ae700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff238067590 0x7ff2380ab8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.522 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.511+0000 7ff23f1ae700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff2380aa1c0 0x7ff23806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.511+0000 7ff23f1ae700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23815a8a0 con 0x7ff2380abe20 2023-06-22T18:07:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.511+0000 7ff23f1ae700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2380ac3c0 con 0x7ff2380aa1c0 2023-06-22T18:07:00.523 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.511+0000 7ff23f1ae700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23815a250 con 0x7ff238067590 2023-06-22T18:07:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.511+0000 7ff23cf4a700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff238067590 0x7ff2380ab8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.511+0000 7ff237fff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff2380abe20 0x7ff2380a9ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:38714/0 (socket says 172.21.15.116:38714) 2023-06-22T18:07:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.511+0000 7ff237fff700 1 -- 172.21.15.116:0/3854208812 learned_addr learned my addr 172.21.15.116:0/3854208812 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-06-22T18:07:00.524 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.512+0000 7ff23d74b700 1 --2- 172.21.15.116:0/3854208812 >> v2:172.21.15.110:3300/0 conn(0x7ff2380aa1c0 0x7ff23806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.512+0000 7ff23cf4a700 1 -- 172.21.15.116:0/3854208812 >> v2:172.21.15.110:3300/0 conn(0x7ff2380aa1c0 msgr2=0x7ff23806e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:00.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.512+0000 7ff23cf4a700 1 --2- 172.21.15.116:0/3854208812 >> v2:172.21.15.110:3300/0 conn(0x7ff2380aa1c0 0x7ff23806e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.512+0000 7ff23cf4a700 1 -- 172.21.15.116:0/3854208812 >> v1:172.21.15.116:6789/0 conn(0x7ff2380abe20 legacy=0x7ff2380a9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.512+0000 7ff2377fe700 1 -- 172.21.15.116:0/3854208812 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262187615 0 0) 0x7ff23815a8a0 con 0x7ff2380abe20 2023-06-22T18:07:00.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.512+0000 7ff23cf4a700 1 -- 172.21.15.116:0/3854208812 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23815a530 con 0x7ff238067590 2023-06-22T18:07:00.525 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.512+0000 7ff23d74b700 1 --2- 172.21.15.116:0/3854208812 >> v2:172.21.15.110:3300/0 conn(0x7ff2380aa1c0 0x7ff23806e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T18:07:00.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.512+0000 7ff23cf4a700 1 --2- 172.21.15.116:0/3854208812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff238067590 0x7ff2380ab8e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff238068420 tx=0x7ff22c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5f9806486d5f686 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:00.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.512+0000 7ff2377fe700 1 -- 172.21.15.116:0/3854208812 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c00b6b0 con 0x7ff238067590 2023-06-22T18:07:00.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.512+0000 7ff2377fe700 1 -- 172.21.15.116:0/3854208812 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22c00b810 con 0x7ff238067590 2023-06-22T18:07:00.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.512+0000 7ff2377fe700 1 -- 172.21.15.116:0/3854208812 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c00bbb0 con 0x7ff238067590 2023-06-22T18:07:00.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.513+0000 7ff23f1ae700 1 -- 172.21.15.116:0/3854208812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff238067590 msgr2=0x7ff2380ab8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.513+0000 7ff23f1ae700 1 --2- 172.21.15.116:0/3854208812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff238067590 0x7ff2380ab8e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff238068420 tx=0x7ff22c0091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.526 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.513+0000 7ff23f1ae700 1 -- 172.21.15.116:0/3854208812 shutdown_connections 2023-06-22T18:07:00.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.513+0000 7ff23f1ae700 1 --2- 172.21.15.116:0/3854208812 >> v2:172.21.15.110:3300/0 conn(0x7ff2380aa1c0 0x7ff23806e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.513+0000 7ff23f1ae700 1 --2- 172.21.15.116:0/3854208812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff238067590 0x7ff2380ab8e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.513+0000 7ff23f1ae700 1 -- 172.21.15.116:0/3854208812 >> 172.21.15.116:0/3854208812 conn(0x7ff23806bbb0 msgr2=0x7ff238068450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:00.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.513+0000 7ff23f1ae700 1 -- 172.21.15.116:0/3854208812 shutdown_connections 2023-06-22T18:07:00.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.513+0000 7ff23f1ae700 1 -- 172.21.15.116:0/3854208812 wait complete. 2023-06-22T18:07:00.527 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.514+0000 7ff23f1ae700 1 Processor -- start 2023-06-22T18:07:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.514+0000 7ff23f1ae700 1 -- start start 2023-06-22T18:07:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23f1ae700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff238067590 0x7ff23814a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23f1ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2380aa1c0 0x7ff23816a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23f1ae700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2380abe20 0x7ff23815e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23f1ae700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23814ce80 con 0x7ff238067590 2023-06-22T18:07:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23f1ae700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380b05d0 con 0x7ff2380aa1c0 2023-06-22T18:07:00.528 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23f1ae700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23816ad00 con 0x7ff2380abe20 2023-06-22T18:07:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23d74b700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2380abe20 0x7ff23815e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23d74b700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2380abe20 0x7ff23815e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48086/0 (socket says 172.21.15.116:48086) 2023-06-22T18:07:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23d74b700 1 -- 172.21.15.116:0/1687217437 learned_addr learned my addr 172.21.15.116:0/1687217437 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-06-22T18:07:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff237fff700 1 --2- 172.21.15.116:0/1687217437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2380aa1c0 0x7ff23816a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23cf4a700 1 --2- 172.21.15.116:0/1687217437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff238067590 0x7ff23814a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23d74b700 1 -- 172.21.15.116:0/1687217437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2380aa1c0 msgr2=0x7ff23816a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.529 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23d74b700 1 --2- 172.21.15.116:0/1687217437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2380aa1c0 0x7ff23816a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23d74b700 1 -- 172.21.15.116:0/1687217437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff238067590 msgr2=0x7ff23814a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23d74b700 1 --2- 172.21.15.116:0/1687217437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff238067590 0x7ff23814a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23d74b700 1 -- 172.21.15.116:0/1687217437 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22c009420 con 0x7ff2380abe20 2023-06-22T18:07:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff237fff700 1 --2- 172.21.15.116:0/1687217437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2380aa1c0 0x7ff23816a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.515+0000 7ff23cf4a700 1 --2- 172.21.15.116:0/1687217437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff238067590 0x7ff23814a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.516+0000 7ff23d74b700 1 --2- 172.21.15.116:0/1687217437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2380abe20 0x7ff23815e0a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff23000d180 tx=0x7ff230009a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:00.530 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.516+0000 7ff234ff9700 1 -- 172.21.15.116:0/1687217437 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23000dcc0 con 0x7ff2380abe20 2023-06-22T18:07:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.516+0000 7ff234ff9700 1 -- 172.21.15.116:0/1687217437 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23000f040 con 0x7ff2380abe20 2023-06-22T18:07:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.516+0000 7ff23f1ae700 1 -- 172.21.15.116:0/1687217437 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2380b2ad0 con 0x7ff2380abe20 2023-06-22T18:07:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.516+0000 7ff234ff9700 1 -- 172.21.15.116:0/1687217437 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff230015690 con 0x7ff2380abe20 2023-06-22T18:07:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.516+0000 7ff23f1ae700 1 -- 172.21.15.116:0/1687217437 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2380aff80 con 0x7ff2380abe20 2023-06-22T18:07:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.516+0000 7ff234ff9700 1 -- 172.21.15.116:0/1687217437 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff230015a80 con 0x7ff2380abe20 2023-06-22T18:07:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.516+0000 7ff234ff9700 1 -- 172.21.15.116:0/1687217437 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7ff23000de40 con 0x7ff2380abe20 2023-06-22T18:07:00.531 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.517+0000 7ff23f1ae700 1 -- 172.21.15.116:0/1687217437 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2380b0780 con 0x7ff2380abe20 2023-06-22T18:07:00.532 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:00.514+0000 7fb312f44180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-22T18:07:00.532 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.519+0000 7ff234ff9700 1 -- 172.21.15.116:0/1687217437 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7ff2380b0780 con 0x7ff2380abe20 2023-06-22T18:07:00.579 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:00.575+0000 7f0696b21180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-22T18:07:00.606 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.602+0000 7ff23f1ae700 1 -- 172.21.15.116:0/1687217437 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd new", "uuid": "89608384-c8bd-4032-98e6-6dd77f29f4de", "id": 2} v 0) v1 -- 0x7ff23814a7f0 con 0x7ff2380abe20 2023-06-22T18:07:00.609 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.605+0000 7ff234ff9700 1 -- 172.21.15.116:0/1687217437 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "89608384-c8bd-4032-98e6-6dd77f29f4de", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff23001a070 con 0x7ff2380abe20 2023-06-22T18:07:00.609 INFO:teuthology.orchestra.run.smithi116.stdout:2 2023-06-22T18:07:00.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.606+0000 7ff23f1ae700 1 -- 172.21.15.116:0/1687217437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2380abe20 msgr2=0x7ff23815e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.606+0000 7ff23f1ae700 1 --2- 172.21.15.116:0/1687217437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2380abe20 0x7ff23815e0a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff23000d180 tx=0x7ff230009a50 comp rx=0 tx=0).stop 2023-06-22T18:07:00.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.606+0000 7ff23f1ae700 1 -- 172.21.15.116:0/1687217437 shutdown_connections 2023-06-22T18:07:00.611 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.606+0000 7ff23f1ae700 1 --2- 172.21.15.116:0/1687217437 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7ff2380abe20 0x7ff23815e0a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.606+0000 7ff23f1ae700 1 --2- 172.21.15.116:0/1687217437 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff238067590 0x7ff23814a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.606+0000 7ff23f1ae700 1 --2- 172.21.15.116:0/1687217437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2380aa1c0 0x7ff23816a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.612 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.606+0000 7ff23f1ae700 1 -- 172.21.15.116:0/1687217437 >> 172.21.15.116:0/1687217437 conn(0x7ff23806bbb0 msgr2=0x7ff23814c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:00.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.606+0000 7ff23f1ae700 1 -- 172.21.15.116:0/1687217437 shutdown_connections 2023-06-22T18:07:00.613 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.606+0000 7ff23f1ae700 1 -- 172.21.15.116:0/1687217437 wait complete. 2023-06-22T18:07:00.622 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new e20af00e-ab57-4686-87b0-a60c90128569 3 2023-06-22T18:07:00.650 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:00.647+0000 7fb312f44180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-22T18:07:00.719 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:00.716+0000 7f0696b21180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-22T18:07:00.725 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:00.722+0000 7fb312f44180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-22T18:07:00.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.736+0000 7f2e275d6700 1 Processor -- start 2023-06-22T18:07:00.740 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.736+0000 7f2e275d6700 1 -- start start 2023-06-22T18:07:00.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e275d6700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2e200aed30 0x7f2e200c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e275d6700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e200afe30 0x7f2e2006db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e275d6700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e200b1620 con 0x7f2e200af100 2023-06-22T18:07:00.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e275d6700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e200b2460 con 0x7f2e200aed30 2023-06-22T18:07:00.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e275d6700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e200b1850 con 0x7f2e200afe30 2023-06-22T18:07:00.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e25b73700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e200afe30 0x7f2e2006db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e25b73700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e200afe30 0x7f2e2006db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38824/0 (socket says 172.21.15.116:38824) 2023-06-22T18:07:00.743 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e25b73700 1 -- 172.21.15.116:0/796820617 learned_addr learned my addr 172.21.15.116:0/796820617 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-06-22T18:07:00.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e25372700 1 --2- 172.21.15.116:0/796820617 >> v2:172.21.15.110:3300/0 conn(0x7f2e200aed30 0x7f2e200c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e25b73700 1 -- 172.21.15.116:0/796820617 >> v2:172.21.15.110:3300/0 conn(0x7f2e200aed30 msgr2=0x7f2e200c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:00.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e25b73700 1 --2- 172.21.15.116:0/796820617 >> v2:172.21.15.110:3300/0 conn(0x7f2e200aed30 0x7f2e200c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e25b73700 1 -- 172.21.15.116:0/796820617 >> v1:172.21.15.110:6789/0 conn(0x7f2e200af100 legacy=0x7f2e200af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e25b73700 1 -- 172.21.15.116:0/796820617 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e200b1b30 con 0x7f2e200afe30 2023-06-22T18:07:00.744 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e25b73700 1 --2- 172.21.15.116:0/796820617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200afe30 0x7f2e2006db10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2e200695a0 tx=0x7f2e18009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a63e2a815dbb0400 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:00.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e17fff700 1 -- 172.21.15.116:0/796820617 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1800e070 con 0x7f2e200afe30 2023-06-22T18:07:00.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e17fff700 1 -- 172.21.15.116:0/796820617 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1800ba20 con 0x7f2e200afe30 2023-06-22T18:07:00.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.737+0000 7f2e17fff700 1 -- 172.21.15.116:0/796820617 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e180092e0 con 0x7f2e200afe30 2023-06-22T18:07:00.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.738+0000 7f2e275d6700 1 -- 172.21.15.116:0/796820617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200afe30 msgr2=0x7f2e2006db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.738+0000 7f2e275d6700 1 --2- 172.21.15.116:0/796820617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200afe30 0x7f2e2006db10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2e200695a0 tx=0x7f2e18009840 comp rx=0 tx=0).stop 2023-06-22T18:07:00.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.738+0000 7f2e275d6700 1 -- 172.21.15.116:0/796820617 shutdown_connections 2023-06-22T18:07:00.745 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.738+0000 7f2e275d6700 1 --2- 172.21.15.116:0/796820617 >> v2:172.21.15.110:3300/0 conn(0x7f2e200aed30 0x7f2e200c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.738+0000 7f2e275d6700 1 --2- 172.21.15.116:0/796820617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200afe30 0x7f2e2006db10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.738+0000 7f2e275d6700 1 -- 172.21.15.116:0/796820617 >> 172.21.15.116:0/796820617 conn(0x7f2e200cd2d0 msgr2=0x7f2e200c92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:00.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.738+0000 7f2e275d6700 1 -- 172.21.15.116:0/796820617 shutdown_connections 2023-06-22T18:07:00.746 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.738+0000 7f2e275d6700 1 -- 172.21.15.116:0/796820617 wait complete. 2023-06-22T18:07:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.739+0000 7f2e275d6700 1 Processor -- start 2023-06-22T18:07:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.739+0000 7f2e275d6700 1 -- start start 2023-06-22T18:07:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e275d6700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e200aed30 0x7f2e20158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e275d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200af100 0x7f2e20156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.748 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e275d6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e200afe30 0x7f2e20159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e275d6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e201588b0 con 0x7f2e200afe30 2023-06-22T18:07:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e275d6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e20158b90 con 0x7f2e200af100 2023-06-22T18:07:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e275d6700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e20158e70 con 0x7f2e200aed30 2023-06-22T18:07:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e25372700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e200aed30 0x7f2e20158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e24b71700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200af100 0x7f2e20156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e25372700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e200aed30 0x7f2e20158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48096/0 (socket says 172.21.15.116:48096) 2023-06-22T18:07:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e25372700 1 -- 172.21.15.116:0/2649669374 learned_addr learned my addr 172.21.15.116:0/2649669374 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-06-22T18:07:00.749 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e25b73700 1 --2- 172.21.15.116:0/2649669374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e200afe30 0x7f2e20159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e24b71700 1 -- 172.21.15.116:0/2649669374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e200aed30 msgr2=0x7f2e20158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e24b71700 1 --2- 172.21.15.116:0/2649669374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e200aed30 0x7f2e20158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e24b71700 1 -- 172.21.15.116:0/2649669374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e200afe30 msgr2=0x7f2e20159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e24b71700 1 --2- 172.21.15.116:0/2649669374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e200afe30 0x7f2e20159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e24b71700 1 -- 172.21.15.116:0/2649669374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18009ae0 con 0x7f2e200af100 2023-06-22T18:07:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.740+0000 7f2e24b71700 1 --2- 172.21.15.116:0/2649669374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200af100 0x7f2e20156160 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00c590 tx=0x7f2e1c007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.741+0000 7f2e157fa700 1 -- 172.21.15.116:0/2649669374 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c011590 con 0x7f2e200af100 2023-06-22T18:07:00.750 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.741+0000 7f2e157fa700 1 -- 172.21.15.116:0/2649669374 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c011710 con 0x7f2e200af100 2023-06-22T18:07:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.741+0000 7f2e157fa700 1 -- 172.21.15.116:0/2649669374 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c011ad0 con 0x7f2e200af100 2023-06-22T18:07:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.741+0000 7f2e275d6700 1 -- 172.21.15.116:0/2649669374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e20159150 con 0x7f2e200af100 2023-06-22T18:07:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.741+0000 7f2e275d6700 1 -- 172.21.15.116:0/2649669374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e201585a0 con 0x7f2e200af100 2023-06-22T18:07:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.741+0000 7f2e157fa700 1 -- 172.21.15.116:0/2649669374 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2e1c01c6c0 con 0x7f2e200af100 2023-06-22T18:07:00.751 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.741+0000 7f2e157fa700 1 -- 172.21.15.116:0/2649669374 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f2e1c0128c0 con 0x7f2e200af100 2023-06-22T18:07:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.742+0000 7f2e275d6700 1 -- 172.21.15.116:0/2649669374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e201595f0 con 0x7f2e200af100 2023-06-22T18:07:00.752 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.744+0000 7f2e157fa700 1 -- 172.21.15.116:0/2649669374 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f2e1c017070 con 0x7f2e200af100 2023-06-22T18:07:00.805 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:00.801+0000 7f0696b21180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-22T18:07:00.839 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.835+0000 7f2e275d6700 1 -- 172.21.15.116:0/2649669374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e20af00e-ab57-4686-87b0-a60c90128569", "id": 3} v 0) v1 -- 0x7f2e2004c1b0 con 0x7f2e200af100 2023-06-22T18:07:00.842 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.838+0000 7f2e157fa700 1 -- 172.21.15.116:0/2649669374 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e20af00e-ab57-4686-87b0-a60c90128569", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2e1c01cba0 con 0x7f2e200af100 2023-06-22T18:07:00.843 INFO:teuthology.orchestra.run.smithi116.stdout:3 2023-06-22T18:07:00.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.839+0000 7f2e275d6700 1 -- 172.21.15.116:0/2649669374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200af100 msgr2=0x7f2e20156160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.839+0000 7f2e275d6700 1 --2- 172.21.15.116:0/2649669374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200af100 0x7f2e20156160 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c00c590 tx=0x7f2e1c007ec0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.845 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.839+0000 7f2e275d6700 1 -- 172.21.15.116:0/2649669374 shutdown_connections 2023-06-22T18:07:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.839+0000 7f2e275d6700 1 --2- 172.21.15.116:0/2649669374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e200afe30 0x7f2e20159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.839+0000 7f2e275d6700 1 --2- 172.21.15.116:0/2649669374 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f2e200aed30 0x7f2e20158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.839+0000 7f2e275d6700 1 --2- 172.21.15.116:0/2649669374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200af100 0x7f2e20156160 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.839+0000 7f2e275d6700 1 -- 172.21.15.116:0/2649669374 >> 172.21.15.116:0/2649669374 conn(0x7f2e200cd2d0 msgr2=0x7f2e200c3920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:00.846 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.840+0000 7f2e275d6700 1 -- 172.21.15.116:0/2649669374 shutdown_connections 2023-06-22T18:07:00.847 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.840+0000 7f2e275d6700 1 -- 172.21.15.116:0/2649669374 wait complete. 2023-06-22T18:07:00.856 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new a9142dbb-15ab-43a0-8501-5ca261a415ef 4 2023-06-22T18:07:00.911 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:00.907+0000 7fb312f44180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-22T18:07:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.967+0000 7f4c58ca3700 1 Processor -- start 2023-06-22T18:07:00.971 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.967+0000 7f4c58ca3700 1 -- start start 2023-06-22T18:07:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c58ca3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4c54157260 0x7f4c5415b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c58ca3700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 0x7f4c5406b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c58ca3700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c540af8c0 con 0x7f4c540ca440 2023-06-22T18:07:00.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c58ca3700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c540b0b90 con 0x7f4c54157260 2023-06-22T18:07:00.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c58ca3700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540b0700 con 0x7f4c54157630 2023-06-22T18:07:00.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c51d9b700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 0x7f4c5406b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c52d9d700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4c540ca440 0x7f4c5406ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:38720/0 (socket says 172.21.15.116:38720) 2023-06-22T18:07:00.973 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c52d9d700 1 -- 172.21.15.116:0/3636711215 learned_addr learned my addr 172.21.15.116:0/3636711215 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-06-22T18:07:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c5259c700 1 --2- 172.21.15.116:0/3636711215 >> v2:172.21.15.110:3300/0 conn(0x7f4c54157260 0x7f4c5415b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c5159a700 1 -- 172.21.15.116:0/3636711215 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099070399 0 0) 0x7f4c540af8c0 con 0x7f4c540ca440 2023-06-22T18:07:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c5159a700 1 -- 172.21.15.116:0/3636711215 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c3c003570 con 0x7f4c540ca440 2023-06-22T18:07:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c51d9b700 1 -- 172.21.15.116:0/3636711215 >> v2:172.21.15.110:3300/0 conn(0x7f4c54157260 msgr2=0x7f4c5415b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c51d9b700 1 --2- 172.21.15.116:0/3636711215 >> v2:172.21.15.110:3300/0 conn(0x7f4c54157260 0x7f4c5415b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c51d9b700 1 -- 172.21.15.116:0/3636711215 >> v1:172.21.15.116:6789/0 conn(0x7f4c540ca440 legacy=0x7f4c5406ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c51d9b700 1 -- 172.21.15.116:0/3636711215 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5415bf00 con 0x7f4c54157630 2023-06-22T18:07:00.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c51d9b700 1 --2- 172.21.15.116:0/3636711215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 0x7f4c5406b610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4c480098e0 tx=0x7f4c480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7d022d503ffc9e5 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:00.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c5159a700 1 -- 172.21.15.116:0/3636711215 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48011630 con 0x7f4c54157630 2023-06-22T18:07:00.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c5159a700 1 -- 172.21.15.116:0/3636711215 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c48011790 con 0x7f4c54157630 2023-06-22T18:07:00.975 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.968+0000 7f4c5159a700 1 -- 172.21.15.116:0/3636711215 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c48011b30 con 0x7f4c54157630 2023-06-22T18:07:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.969+0000 7f4c58ca3700 1 -- 172.21.15.116:0/3636711215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 msgr2=0x7f4c5406b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.969+0000 7f4c58ca3700 1 --2- 172.21.15.116:0/3636711215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 0x7f4c5406b610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4c480098e0 tx=0x7f4c480091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.969+0000 7f4c58ca3700 1 -- 172.21.15.116:0/3636711215 shutdown_connections 2023-06-22T18:07:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.969+0000 7f4c58ca3700 1 --2- 172.21.15.116:0/3636711215 >> v2:172.21.15.110:3300/0 conn(0x7f4c54157260 0x7f4c5415b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.969+0000 7f4c58ca3700 1 --2- 172.21.15.116:0/3636711215 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 0x7f4c5406b610 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.969+0000 7f4c58ca3700 1 -- 172.21.15.116:0/3636711215 >> 172.21.15.116:0/3636711215 conn(0x7f4c540ce210 msgr2=0x7f4c540ce610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.969+0000 7f4c58ca3700 1 -- 172.21.15.116:0/3636711215 shutdown_connections 2023-06-22T18:07:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.969+0000 7f4c58ca3700 1 -- 172.21.15.116:0/3636711215 wait complete. 2023-06-22T18:07:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.970+0000 7f4c58ca3700 1 Processor -- start 2023-06-22T18:07:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.970+0000 7f4c58ca3700 1 -- start start 2023-06-22T18:07:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c58ca3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c540ca440 0x7f4c5414f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c58ca3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c54157260 0x7f4c5416b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c58ca3700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 0x7f4c54153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c58ca3700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5415a6e0 con 0x7f4c54157260 2023-06-22T18:07:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c58ca3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5414f740 con 0x7f4c540ca440 2023-06-22T18:07:00.979 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c58ca3700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5416b960 con 0x7f4c54157630 2023-06-22T18:07:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c5259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c540ca440 0x7f4c5414f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c52d9d700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 0x7f4c54153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c5259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c540ca440 0x7f4c5414f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38844/0 (socket says 172.21.15.116:38844) 2023-06-22T18:07:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c5259c700 1 -- 172.21.15.116:0/2733965497 learned_addr learned my addr 172.21.15.116:0/2733965497 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-06-22T18:07:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c51d9b700 1 --2- 172.21.15.116:0/2733965497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c54157260 0x7f4c5416b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c52d9d700 1 -- 172.21.15.116:0/2733965497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c540ca440 msgr2=0x7f4c5414f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.980 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c52d9d700 1 --2- 172.21.15.116:0/2733965497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c540ca440 0x7f4c5414f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c52d9d700 1 -- 172.21.15.116:0/2733965497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c54157260 msgr2=0x7f4c5416b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c52d9d700 1 --2- 172.21.15.116:0/2733965497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c54157260 0x7f4c5416b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c52d9d700 1 -- 172.21.15.116:0/2733965497 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c48009420 con 0x7f4c54157630 2023-06-22T18:07:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c5259c700 1 --2- 172.21.15.116:0/2733965497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c540ca440 0x7f4c5414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c52d9d700 1 --2- 172.21.15.116:0/2733965497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 0x7f4c54153990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00cb90 tx=0x7f4c4c001980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c427fc700 1 -- 172.21.15.116:0/2733965497 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00a7e0 con 0x7f4c54157630 2023-06-22T18:07:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c427fc700 1 -- 172.21.15.116:0/2733965497 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c00a940 con 0x7f4c54157630 2023-06-22T18:07:00.981 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c58ca3700 1 -- 172.21.15.116:0/2733965497 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c54158b10 con 0x7f4c54157630 2023-06-22T18:07:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c427fc700 1 -- 172.21.15.116:0/2733965497 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00acc0 con 0x7f4c54157630 2023-06-22T18:07:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c58ca3700 1 -- 172.21.15.116:0/2733965497 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c541581e0 con 0x7f4c54157630 2023-06-22T18:07:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c427fc700 1 -- 172.21.15.116:0/2733965497 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4c4c007580 con 0x7f4c54157630 2023-06-22T18:07:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.972+0000 7f4c427fc700 1 -- 172.21.15.116:0/2733965497 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f4c4c01ca40 con 0x7f4c54157630 2023-06-22T18:07:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.973+0000 7f4c58ca3700 1 -- 172.21.15.116:0/2733965497 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c54158ca0 con 0x7f4c54157630 2023-06-22T18:07:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:00.975+0000 7f4c427fc700 1 -- 172.21.15.116:0/2733965497 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f4c4c021060 con 0x7f4c54157630 2023-06-22T18:07:00.997 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:00.993+0000 7f0696b21180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-22T18:07:01.061 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.057+0000 7f4c58ca3700 1 -- 172.21.15.116:0/2733965497 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a9142dbb-15ab-43a0-8501-5ca261a415ef", "id": 4} v 0) v1 -- 0x7f4c5404c1b0 con 0x7f4c54157630 2023-06-22T18:07:01.064 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.060+0000 7f4c427fc700 1 -- 172.21.15.116:0/2733965497 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a9142dbb-15ab-43a0-8501-5ca261a415ef", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4c4c02a0b0 con 0x7f4c54157630 2023-06-22T18:07:01.065 INFO:teuthology.orchestra.run.smithi116.stdout:4 2023-06-22T18:07:01.065 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.061+0000 7f4c58ca3700 1 -- 172.21.15.116:0/2733965497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 msgr2=0x7f4c54153990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.061+0000 7f4c58ca3700 1 --2- 172.21.15.116:0/2733965497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 0x7f4c54153990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00cb90 tx=0x7f4c4c001980 comp rx=0 tx=0).stop 2023-06-22T18:07:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.061+0000 7f4c58ca3700 1 -- 172.21.15.116:0/2733965497 shutdown_connections 2023-06-22T18:07:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.061+0000 7f4c58ca3700 1 --2- 172.21.15.116:0/2733965497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c540ca440 0x7f4c5414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.061+0000 7f4c58ca3700 1 --2- 172.21.15.116:0/2733965497 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f4c54157630 0x7f4c54153990 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.066 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.061+0000 7f4c58ca3700 1 --2- 172.21.15.116:0/2733965497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c54157260 0x7f4c5416b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.061+0000 7f4c58ca3700 1 -- 172.21.15.116:0/2733965497 >> 172.21.15.116:0/2733965497 conn(0x7f4c540ce210 msgr2=0x7f4c540ca900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:01.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.062+0000 7f4c58ca3700 1 -- 172.21.15.116:0/2733965497 shutdown_connections 2023-06-22T18:07:01.067 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.062+0000 7f4c58ca3700 1 -- 172.21.15.116:0/2733965497 wait complete. 2023-06-22T18:07:01.077 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new e81f0aa0-d55d-4d72-937e-a1bc82b3117f 5 2023-06-22T18:07:01.083 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:01.080+0000 7fb312f44180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-22T18:07:01.160 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:01.157+0000 7fb312f44180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-22T18:07:01.177 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:01.173+0000 7f0696b21180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-22T18:07:01.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd46d43700 1 Processor -- start 2023-06-22T18:07:01.192 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd46d43700 1 -- start start 2023-06-22T18:07:01.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd46d43700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd400cf570 0x7fcd400cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.193 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd46d43700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd400b3220 con 0x7fcd4009d7e0 2023-06-22T18:07:01.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd46d43700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd400b37e0 con 0x7fcd4009dbb0 2023-06-22T18:07:01.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd46d43700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd4009e030 con 0x7fcd400cf570 2023-06-22T18:07:01.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd452e0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd400cf570 0x7fcd400cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd3ffff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcd4009dbb0 0x7fcd400cca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:38726/0 (socket says 172.21.15.116:38726) 2023-06-22T18:07:01.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd3ffff700 1 -- 172.21.15.116:0/2910785657 learned_addr learned my addr 172.21.15.116:0/2910785657 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-06-22T18:07:01.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd452e0700 1 -- 172.21.15.116:0/2910785657 >> v1:172.21.15.116:6789/0 conn(0x7fcd4009dbb0 legacy=0x7fcd400cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.194 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd452e0700 1 -- 172.21.15.116:0/2910785657 >> v1:172.21.15.110:6789/0 conn(0x7fcd4009d7e0 legacy=0x7fcd400b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd452e0700 1 -- 172.21.15.116:0/2910785657 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd400cd500 con 0x7fcd400cf570 2023-06-22T18:07:01.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd452e0700 1 --2- 172.21.15.116:0/2910785657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd400cf570 0x7fcd400cf9d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009e60 tx=0x7fcd38009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f20cc95682bd40ec server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:01.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd3f7fe700 1 -- 172.21.15.116:0/2910785657 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3800e070 con 0x7fcd400cf570 2023-06-22T18:07:01.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd3f7fe700 1 -- 172.21.15.116:0/2910785657 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3800bc30 con 0x7fcd400cf570 2023-06-22T18:07:01.195 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.189+0000 7fcd3f7fe700 1 -- 172.21.15.116:0/2910785657 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38012430 con 0x7fcd400cf570 2023-06-22T18:07:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.191+0000 7fcd46d43700 1 -- 172.21.15.116:0/2910785657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd400cf570 msgr2=0x7fcd400cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.191+0000 7fcd46d43700 1 --2- 172.21.15.116:0/2910785657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd400cf570 0x7fcd400cf9d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009e60 tx=0x7fcd38009780 comp rx=0 tx=0).stop 2023-06-22T18:07:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.191+0000 7fcd46d43700 1 -- 172.21.15.116:0/2910785657 shutdown_connections 2023-06-22T18:07:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.191+0000 7fcd46d43700 1 --2- 172.21.15.116:0/2910785657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd400cf570 0x7fcd400cf9d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.191+0000 7fcd46d43700 1 -- 172.21.15.116:0/2910785657 >> 172.21.15.116:0/2910785657 conn(0x7fcd40067340 msgr2=0x7fcd40067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.191+0000 7fcd46d43700 1 -- 172.21.15.116:0/2910785657 shutdown_connections 2023-06-22T18:07:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.191+0000 7fcd46d43700 1 -- 172.21.15.116:0/2910785657 wait complete. 2023-06-22T18:07:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd46d43700 1 Processor -- start 2023-06-22T18:07:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd46d43700 1 -- start start 2023-06-22T18:07:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd46d43700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4009d7e0 0x7fcd40159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd46d43700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd4009dbb0 0x7fcd400c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd46d43700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd400cf570 0x7fcd40153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd46d43700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4015b070 con 0x7fcd400cf570 2023-06-22T18:07:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd46d43700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd400c88d0 con 0x7fcd4009d7e0 2023-06-22T18:07:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd46d43700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4015a3f0 con 0x7fcd4009dbb0 2023-06-22T18:07:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd3ffff700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd4009dbb0 0x7fcd400c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd44adf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4009d7e0 0x7fcd40159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd3ffff700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd4009dbb0 0x7fcd400c8390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48134/0 (socket says 172.21.15.116:48134) 2023-06-22T18:07:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd3ffff700 1 -- 172.21.15.116:0/1903761289 learned_addr learned my addr 172.21.15.116:0/1903761289 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-06-22T18:07:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd452e0700 1 --2- 172.21.15.116:0/1903761289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd400cf570 0x7fcd40153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd3ffff700 1 -- 172.21.15.116:0/1903761289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4009d7e0 msgr2=0x7fcd40159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd3ffff700 1 --2- 172.21.15.116:0/1903761289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4009d7e0 0x7fcd40159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd3ffff700 1 -- 172.21.15.116:0/1903761289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd400cf570 msgr2=0x7fcd40153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd3ffff700 1 --2- 172.21.15.116:0/1903761289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd400cf570 0x7fcd40153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd3ffff700 1 -- 172.21.15.116:0/1903761289 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd38009a20 con 0x7fcd4009dbb0 2023-06-22T18:07:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.192+0000 7fcd44adf700 1 --2- 172.21.15.116:0/1903761289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4009d7e0 0x7fcd40159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.193+0000 7fcd3ffff700 1 --2- 172.21.15.116:0/1903761289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd4009dbb0 0x7fcd400c8390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00c7c0 tx=0x7fcd2c007f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:01.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.193+0000 7fcd3cff9700 1 -- 172.21.15.116:0/1903761289 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c005b30 con 0x7fcd4009dbb0 2023-06-22T18:07:01.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.193+0000 7fcd3cff9700 1 -- 172.21.15.116:0/1903761289 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c005cb0 con 0x7fcd4009dbb0 2023-06-22T18:07:01.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.193+0000 7fcd46d43700 1 -- 172.21.15.116:0/1903761289 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4015a6d0 con 0x7fcd4009dbb0 2023-06-22T18:07:01.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.193+0000 7fcd3cff9700 1 -- 172.21.15.116:0/1903761289 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c004a80 con 0x7fcd4009dbb0 2023-06-22T18:07:01.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.193+0000 7fcd46d43700 1 -- 172.21.15.116:0/1903761289 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4015ac90 con 0x7fcd4009dbb0 2023-06-22T18:07:01.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.193+0000 7fcd3cff9700 1 -- 172.21.15.116:0/1903761289 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fcd2c007350 con 0x7fcd4009dbb0 2023-06-22T18:07:01.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.193+0000 7fcd3cff9700 1 -- 172.21.15.116:0/1903761289 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fcd2c0055b0 con 0x7fcd4009dbb0 2023-06-22T18:07:01.202 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.195+0000 7fcd46d43700 1 -- 172.21.15.116:0/1903761289 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4004c1b0 con 0x7fcd4009dbb0 2023-06-22T18:07:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.197+0000 7fcd3cff9700 1 -- 172.21.15.116:0/1903761289 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fcd2c021060 con 0x7fcd4009dbb0 2023-06-22T18:07:01.258 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:01.255+0000 7f0696b21180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-22T18:07:01.281 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.279+0000 7fcd46d43700 1 -- 172.21.15.116:0/1903761289 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e81f0aa0-d55d-4d72-937e-a1bc82b3117f", "id": 5} v 0) v1 -- 0x7fcd400cd7e0 con 0x7fcd4009dbb0 2023-06-22T18:07:01.285 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.282+0000 7fcd3cff9700 1 -- 172.21.15.116:0/1903761289 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e81f0aa0-d55d-4d72-937e-a1bc82b3117f", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcd2c007600 con 0x7fcd4009dbb0 2023-06-22T18:07:01.285 INFO:teuthology.orchestra.run.smithi116.stdout:5 2023-06-22T18:07:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.283+0000 7fcd46d43700 1 -- 172.21.15.116:0/1903761289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd4009dbb0 msgr2=0x7fcd400c8390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.283+0000 7fcd46d43700 1 --2- 172.21.15.116:0/1903761289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd4009dbb0 0x7fcd400c8390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c00c7c0 tx=0x7fcd2c007f70 comp rx=0 tx=0).stop 2023-06-22T18:07:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.283+0000 7fcd46d43700 1 -- 172.21.15.116:0/1903761289 shutdown_connections 2023-06-22T18:07:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.283+0000 7fcd46d43700 1 --2- 172.21.15.116:0/1903761289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd4009d7e0 0x7fcd40159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.283+0000 7fcd46d43700 1 --2- 172.21.15.116:0/1903761289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd400cf570 0x7fcd40153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.283+0000 7fcd46d43700 1 --2- 172.21.15.116:0/1903761289 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcd4009dbb0 0x7fcd400c8390 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.283+0000 7fcd46d43700 1 -- 172.21.15.116:0/1903761289 >> 172.21.15.116:0/1903761289 conn(0x7fcd40067340 msgr2=0x7fcd40158b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.283+0000 7fcd46d43700 1 -- 172.21.15.116:0/1903761289 shutdown_connections 2023-06-22T18:07:01.288 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.283+0000 7fcd46d43700 1 -- 172.21.15.116:0/1903761289 wait complete. 2023-06-22T18:07:01.294 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:01.291+0000 7fb312f44180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-22T18:07:01.299 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 25f64c11-e6e8-46d1-b064-28743c4233df 6 2023-06-22T18:07:01.391 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:01.388+0000 7fb312f44180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-22T18:07:01.398 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:01.396+0000 7f0696b21180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-22T18:07:01.417 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.414+0000 7fd872261700 1 Processor -- start 2023-06-22T18:07:01.418 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.414+0000 7fd872261700 1 -- start start 2023-06-22T18:07:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd872261700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd86c13fa70 0x7fd86c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd872261700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 0x7fd86c06b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd872261700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86c06ea00 con 0x7fd86c0cad20 2023-06-22T18:07:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd872261700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c06ece0 con 0x7fd86c13fa70 2023-06-22T18:07:01.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd872261700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c06efc0 con 0x7fd86c13fe40 2023-06-22T18:07:01.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86b7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd86c13fa70 0x7fd86c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86affd700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 0x7fd86c06b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.422 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86bfff700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd86c0cad20 0x7fd86c0c9ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:38730/0 (socket says 172.21.15.116:38730) 2023-06-22T18:07:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86b7fe700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd86c13fa70 0x7fd86c06e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38890/0 (socket says 172.21.15.116:38890) 2023-06-22T18:07:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86bfff700 1 -- 172.21.15.116:0/1869104457 learned_addr learned my addr 172.21.15.116:0/1869104457 (peer_addr_for_me v1:172.21.15.116:0/0) 2023-06-22T18:07:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86b7fe700 1 -- 172.21.15.116:0/1869104457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 msgr2=0x7fd86c06b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86b7fe700 1 --2- 172.21.15.116:0/1869104457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 0x7fd86c06b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86b7fe700 1 -- 172.21.15.116:0/1869104457 >> v1:172.21.15.116:6789/0 conn(0x7fd86c0cad20 legacy=0x7fd86c0c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86a7fc700 1 -- 172.21.15.116:0/1869104457 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940572021 0 0) 0x7fd86c06ea00 con 0x7fd86c0cad20 2023-06-22T18:07:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86b7fe700 1 -- 172.21.15.116:0/1869104457 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c06f2a0 con 0x7fd86c13fa70 2023-06-22T18:07:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86affd700 1 --2- 172.21.15.116:0/1869104457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 0x7fd86c06b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.415+0000 7fd86b7fe700 1 --2- 172.21.15.116:0/1869104457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c13fa70 0x7fd86c06e420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd8600098e0 tx=0x7fd8600091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=868c73aa43cbc2b2 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:01.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.416+0000 7fd86a7fc700 1 -- 172.21.15.116:0/1869104457 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86000b640 con 0x7fd86c13fa70 2023-06-22T18:07:01.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.416+0000 7fd86a7fc700 1 -- 172.21.15.116:0/1869104457 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86000b7a0 con 0x7fd86c13fa70 2023-06-22T18:07:01.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.416+0000 7fd86a7fc700 1 -- 172.21.15.116:0/1869104457 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86000bb40 con 0x7fd86c13fa70 2023-06-22T18:07:01.426 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.416+0000 7fd872261700 1 -- 172.21.15.116:0/1869104457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c13fa70 msgr2=0x7fd86c06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.416+0000 7fd872261700 1 --2- 172.21.15.116:0/1869104457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c13fa70 0x7fd86c06e420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd8600098e0 tx=0x7fd8600091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.416+0000 7fd872261700 1 -- 172.21.15.116:0/1869104457 shutdown_connections 2023-06-22T18:07:01.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.416+0000 7fd872261700 1 --2- 172.21.15.116:0/1869104457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c13fa70 0x7fd86c06e420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.416+0000 7fd872261700 1 --2- 172.21.15.116:0/1869104457 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 0x7fd86c06b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.416+0000 7fd872261700 1 -- 172.21.15.116:0/1869104457 >> 172.21.15.116:0/1869104457 conn(0x7fd86c068450 msgr2=0x7fd86c068850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:01.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.416+0000 7fd872261700 1 -- 172.21.15.116:0/1869104457 shutdown_connections 2023-06-22T18:07:01.427 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.416+0000 7fd872261700 1 -- 172.21.15.116:0/1869104457 wait complete. 2023-06-22T18:07:01.429 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.418+0000 7fd872261700 1 Processor -- start 2023-06-22T18:07:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.418+0000 7fd872261700 1 -- start start 2023-06-22T18:07:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd872261700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd86c0cad20 0x7fd86c159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd872261700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c13fa70 0x7fd86c13d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd872261700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 0x7fd86c13d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd872261700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c13cc90 con 0x7fd86c0cad20 2023-06-22T18:07:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd872261700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0c4200 con 0x7fd86c13fa70 2023-06-22T18:07:01.430 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd872261700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0c44e0 con 0x7fd86c13fe40 2023-06-22T18:07:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c13fa70 0x7fd86c13d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86bfff700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 0x7fd86c13d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c13fa70 0x7fd86c13d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38898/0 (socket says 172.21.15.116:38898) 2023-06-22T18:07:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86affd700 1 -- 172.21.15.116:0/1866556534 learned_addr learned my addr 172.21.15.116:0/1866556534 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-06-22T18:07:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86b7fe700 1 --2- 172.21.15.116:0/1866556534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd86c0cad20 0x7fd86c159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86affd700 1 -- 172.21.15.116:0/1866556534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 msgr2=0x7fd86c13d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86affd700 1 --2- 172.21.15.116:0/1866556534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 0x7fd86c13d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.431 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86affd700 1 -- 172.21.15.116:0/1866556534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd86c0cad20 msgr2=0x7fd86c159140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86affd700 1 --2- 172.21.15.116:0/1866556534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd86c0cad20 0x7fd86c159140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86affd700 1 -- 172.21.15.116:0/1866556534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c009420 con 0x7fd86c13fa70 2023-06-22T18:07:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86bfff700 1 --2- 172.21.15.116:0/1866556534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 0x7fd86c13d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd86affd700 1 --2- 172.21.15.116:0/1866556534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c13fa70 0x7fd86c13d1c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00ecb0 tx=0x7fd85c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd8537fe700 1 -- 172.21.15.116:0/1866556534 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c00c850 con 0x7fd86c13fa70 2023-06-22T18:07:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd8537fe700 1 -- 172.21.15.116:0/1866556534 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c00c9b0 con 0x7fd86c13fa70 2023-06-22T18:07:01.432 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd872261700 1 -- 172.21.15.116:0/1866556534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd860009420 con 0x7fd86c13fa70 2023-06-22T18:07:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd8537fe700 1 -- 172.21.15.116:0/1866556534 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c00cd30 con 0x7fd86c13fa70 2023-06-22T18:07:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.420+0000 7fd872261700 1 -- 172.21.15.116:0/1866556534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c0c8db0 con 0x7fd86c13fa70 2023-06-22T18:07:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.421+0000 7fd8537fe700 1 -- 172.21.15.116:0/1866556534 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd85c005c80 con 0x7fd86c13fa70 2023-06-22T18:07:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.421+0000 7fd8537fe700 1 -- 172.21.15.116:0/1866556534 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fd85c006620 con 0x7fd86c13fa70 2023-06-22T18:07:01.433 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.421+0000 7fd872261700 1 -- 172.21.15.116:0/1866556534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c0c4650 con 0x7fd86c13fa70 2023-06-22T18:07:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.423+0000 7fd8537fe700 1 -- 172.21.15.116:0/1866556534 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fd85c021060 con 0x7fd86c13fa70 2023-06-22T18:07:01.500 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:01.497+0000 7f0696b21180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-22T18:07:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.506+0000 7fd872261700 1 -- 172.21.15.116:0/1866556534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "25f64c11-e6e8-46d1-b064-28743c4233df", "id": 6} v 0) v1 -- 0x7fd86c04c1b0 con 0x7fd86c13fa70 2023-06-22T18:07:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.514+0000 7fd8537fe700 1 -- 172.21.15.116:0/1866556534 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "25f64c11-e6e8-46d1-b064-28743c4233df", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd85c017070 con 0x7fd86c13fa70 2023-06-22T18:07:01.517 INFO:teuthology.orchestra.run.smithi116.stdout:6 2023-06-22T18:07:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.515+0000 7fd872261700 1 -- 172.21.15.116:0/1866556534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c13fa70 msgr2=0x7fd86c13d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.515+0000 7fd872261700 1 --2- 172.21.15.116:0/1866556534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c13fa70 0x7fd86c13d1c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00ecb0 tx=0x7fd85c0091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.515+0000 7fd872261700 1 -- 172.21.15.116:0/1866556534 shutdown_connections 2023-06-22T18:07:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.515+0000 7fd872261700 1 --2- 172.21.15.116:0/1866556534 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd86c13fe40 0x7fd86c13d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.515+0000 7fd872261700 1 --2- 172.21.15.116:0/1866556534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd86c0cad20 0x7fd86c159140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.515+0000 7fd872261700 1 --2- 172.21.15.116:0/1866556534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd86c13fa70 0x7fd86c13d1c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.515+0000 7fd872261700 1 -- 172.21.15.116:0/1866556534 >> 172.21.15.116:0/1866556534 conn(0x7fd86c068450 msgr2=0x7fd86c0c9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.515+0000 7fd872261700 1 -- 172.21.15.116:0/1866556534 shutdown_connections 2023-06-22T18:07:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.515+0000 7fd872261700 1 -- 172.21.15.116:0/1866556534 wait complete. 2023-06-22T18:07:01.530 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd new 608edcd6-65d3-4e76-bbd4-a612a6befa4f 7 2023-06-22T18:07:01.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.642+0000 7f27f61b8700 1 Processor -- start 2023-06-22T18:07:01.645 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.642+0000 7f27f61b8700 1 -- start start 2023-06-22T18:07:01.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27f61b8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f27f009d7e0 0x7f27f009dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27f61b8700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00b3c70 0x7f27f00cca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27f61b8700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f27f00ccf80 0x7f27f00cf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27f61b8700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f00b3220 con 0x7f27f009d7e0 2023-06-22T18:07:01.652 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27f61b8700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f00b37e0 con 0x7f27f00ccf80 2023-06-22T18:07:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27f61b8700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00cd500 con 0x7f27f00b3c70 2023-06-22T18:07:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27eeffd700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00b3c70 0x7f27f00cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27effff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f27f00ccf80 0x7f27f00cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27effff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f27f00ccf80 0x7f27f00cf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38912/0 (socket says 172.21.15.116:38912) 2023-06-22T18:07:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27eeffd700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00b3c70 0x7f27f00cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.116:48154/0 (socket says 172.21.15.116:48154) 2023-06-22T18:07:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27effff700 1 -- 172.21.15.116:0/1708072621 learned_addr learned my addr 172.21.15.116:0/1708072621 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-06-22T18:07:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27ef7fe700 1 --2- 172.21.15.116:0/1708072621 >> v2:172.21.15.110:3300/0 conn(0x7f27f009d7e0 0x7f27f009dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.653 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27eeffd700 1 -- 172.21.15.116:0/1708072621 >> v2:172.21.15.116:3300/0 conn(0x7f27f00ccf80 msgr2=0x7f27f00cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:01.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27eeffd700 1 --2- 172.21.15.116:0/1708072621 >> v2:172.21.15.116:3300/0 conn(0x7f27f00ccf80 0x7f27f00cf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27eeffd700 1 -- 172.21.15.116:0/1708072621 >> v2:172.21.15.110:3300/0 conn(0x7f27f009d7e0 msgr2=0x7f27f009dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:01.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27eeffd700 1 --2- 172.21.15.116:0/1708072621 >> v2:172.21.15.110:3300/0 conn(0x7f27f009d7e0 0x7f27f009dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27eeffd700 1 -- 172.21.15.116:0/1708072621 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f00cd7e0 con 0x7f27f00b3c70 2023-06-22T18:07:01.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27ef7fe700 1 --2- 172.21.15.116:0/1708072621 >> v2:172.21.15.110:3300/0 conn(0x7f27f009d7e0 0x7f27f009dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T18:07:01.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27effff700 1 --2- 172.21.15.116:0/1708072621 >> v2:172.21.15.116:3300/0 conn(0x7f27f00ccf80 0x7f27f00cf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:01.654 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27eeffd700 1 --2- 172.21.15.116:0/1708072621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00b3c70 0x7f27f00cca40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f27f00caa30 tx=0x7f27e00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a9791e19f77e2cb server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27ee7fc700 1 -- 172.21.15.116:0/1708072621 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e000b690 con 0x7f27f00b3c70 2023-06-22T18:07:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27ee7fc700 1 -- 172.21.15.116:0/1708072621 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e000b7f0 con 0x7f27f00b3c70 2023-06-22T18:07:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.643+0000 7f27ee7fc700 1 -- 172.21.15.116:0/1708072621 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e000bb90 con 0x7f27f00b3c70 2023-06-22T18:07:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.644+0000 7f27f61b8700 1 -- 172.21.15.116:0/1708072621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00b3c70 msgr2=0x7f27f00cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.644+0000 7f27f61b8700 1 --2- 172.21.15.116:0/1708072621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00b3c70 0x7f27f00cca40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f27f00caa30 tx=0x7f27e00091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.644+0000 7f27f61b8700 1 -- 172.21.15.116:0/1708072621 shutdown_connections 2023-06-22T18:07:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.644+0000 7f27f61b8700 1 --2- 172.21.15.116:0/1708072621 >> v2:172.21.15.110:3300/0 conn(0x7f27f009d7e0 0x7f27f009dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.655 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.644+0000 7f27f61b8700 1 --2- 172.21.15.116:0/1708072621 >> v2:172.21.15.116:3300/0 conn(0x7f27f00ccf80 0x7f27f00cf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.644+0000 7f27f61b8700 1 --2- 172.21.15.116:0/1708072621 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00b3c70 0x7f27f00cca40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.644+0000 7f27f61b8700 1 -- 172.21.15.116:0/1708072621 >> 172.21.15.116:0/1708072621 conn(0x7f27f0067340 msgr2=0x7f27f0067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:01.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.644+0000 7f27f61b8700 1 -- 172.21.15.116:0/1708072621 shutdown_connections 2023-06-22T18:07:01.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.644+0000 7f27f61b8700 1 -- 172.21.15.116:0/1708072621 wait complete. 2023-06-22T18:07:01.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.645+0000 7f27f61b8700 1 Processor -- start 2023-06-22T18:07:01.656 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.645+0000 7f27f61b8700 1 -- start start 2023-06-22T18:07:01.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27f61b8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f009d7e0 0x7f27f00cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27f61b8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27f00b3c70 0x7f27f00c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27f61b8700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00ccf80 0x7f27f014ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27f61b8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00c97b0 con 0x7f27f009d7e0 2023-06-22T18:07:01.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27f61b8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00cb2b0 con 0x7f27f00b3c70 2023-06-22T18:07:01.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27f61b8700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f00cb590 con 0x7f27f00ccf80 2023-06-22T18:07:01.657 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27eeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27f00b3c70 0x7f27f00c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27eeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27f00b3c70 0x7f27f00c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38920/0 (socket says 172.21.15.116:38920) 2023-06-22T18:07:01.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27eeffd700 1 -- 172.21.15.116:0/1847184861 learned_addr learned my addr 172.21.15.116:0/1847184861 (peer_addr_for_me v2:172.21.15.116:0/0) 2023-06-22T18:07:01.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27effff700 1 --2- 172.21.15.116:0/1847184861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00ccf80 0x7f27f014ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27ef7fe700 1 --2- 172.21.15.116:0/1847184861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f009d7e0 0x7f27f00cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27eeffd700 1 -- 172.21.15.116:0/1847184861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00ccf80 msgr2=0x7f27f014ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27eeffd700 1 --2- 172.21.15.116:0/1847184861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00ccf80 0x7f27f014ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.658 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27eeffd700 1 -- 172.21.15.116:0/1847184861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f009d7e0 msgr2=0x7f27f00cc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27eeffd700 1 --2- 172.21.15.116:0/1847184861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f009d7e0 0x7f27f00cc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27eeffd700 1 -- 172.21.15.116:0/1847184861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e8009ae0 con 0x7f27f00b3c70 2023-06-22T18:07:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27effff700 1 --2- 172.21.15.116:0/1847184861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00ccf80 0x7f27f014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.646+0000 7f27eeffd700 1 --2- 172.21.15.116:0/1847184861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27f00b3c70 0x7f27f00c8180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f27f00cd640 tx=0x7f27e0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.647+0000 7f27d77fe700 1 -- 172.21.15.116:0/1847184861 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0000d10 con 0x7f27f00b3c70 2023-06-22T18:07:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.647+0000 7f27d77fe700 1 -- 172.21.15.116:0/1847184861 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e000bb70 con 0x7f27f00b3c70 2023-06-22T18:07:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.647+0000 7f27d77fe700 1 -- 172.21.15.116:0/1847184861 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0011650 con 0x7f27f00b3c70 2023-06-22T18:07:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.647+0000 7f27f61b8700 1 -- 172.21.15.116:0/1847184861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e0009420 con 0x7f27f00b3c70 2023-06-22T18:07:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.647+0000 7f27f61b8700 1 -- 172.21.15.116:0/1847184861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f00cbe30 con 0x7f27f00b3c70 2023-06-22T18:07:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.647+0000 7f27d77fe700 1 -- 172.21.15.116:0/1847184861 <== mon.1 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f27e0011c80 con 0x7f27f00b3c70 2023-06-22T18:07:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.647+0000 7f27d77fe700 1 -- 172.21.15.116:0/1847184861 <== mon.1 v2:172.21.15.116:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f27e00253f0 con 0x7f27f00b3c70 2023-06-22T18:07:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.647+0000 7f27f61b8700 1 -- 172.21.15.116:0/1847184861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f00cb950 con 0x7f27f00b3c70 2023-06-22T18:07:01.661 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.650+0000 7f27d77fe700 1 -- 172.21.15.116:0/1847184861 <== mon.1 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f27e0033060 con 0x7f27f00b3c70 2023-06-22T18:07:01.733 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.730+0000 7f27f61b8700 1 -- 172.21.15.116:0/1847184861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "608edcd6-65d3-4e76-bbd4-a612a6befa4f", "id": 7} v 0) v1 -- 0x7f27f00cc050 con 0x7f27f00b3c70 2023-06-22T18:07:01.739 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.736+0000 7f27d77fe700 1 -- 172.21.15.116:0/1847184861 <== mon.1 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "608edcd6-65d3-4e76-bbd4-a612a6befa4f", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f27e0015020 con 0x7f27f00b3c70 2023-06-22T18:07:01.740 INFO:teuthology.orchestra.run.smithi116.stdout:7 2023-06-22T18:07:01.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.737+0000 7f27f61b8700 1 -- 172.21.15.116:0/1847184861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27f00b3c70 msgr2=0x7f27f00c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.737+0000 7f27f61b8700 1 --2- 172.21.15.116:0/1847184861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27f00b3c70 0x7f27f00c8180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f27f00cd640 tx=0x7f27e0017000 comp rx=0 tx=0).stop 2023-06-22T18:07:01.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.737+0000 7f27f61b8700 1 -- 172.21.15.116:0/1847184861 shutdown_connections 2023-06-22T18:07:01.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.737+0000 7f27f61b8700 1 --2- 172.21.15.116:0/1847184861 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27f00ccf80 0x7f27f014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.737+0000 7f27f61b8700 1 --2- 172.21.15.116:0/1847184861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27f009d7e0 0x7f27f00cc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.741 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.737+0000 7f27f61b8700 1 --2- 172.21.15.116:0/1847184861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27f00b3c70 0x7f27f00c8180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.737+0000 7f27f61b8700 1 -- 172.21.15.116:0/1847184861 >> 172.21.15.116:0/1847184861 conn(0x7f27f0067340 msgr2=0x7f27f014dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:01.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.737+0000 7f27f61b8700 1 -- 172.21.15.116:0/1847184861 shutdown_connections 2023-06-22T18:07:01.742 INFO:teuthology.orchestra.run.smithi116.stderr:2023-06-22T18:07:01.737+0000 7f27f61b8700 1 -- 172.21.15.116:0/1847184861 wait complete. 2023-06-22T18:07:01.752 INFO:tasks.ceph.osd.0:Restarting daemon 2023-06-22T18:07:01.752 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-06-22T18:07:01.754 INFO:tasks.ceph.osd.0:Started 2023-06-22T18:07:01.754 INFO:tasks.ceph.osd.1:Restarting daemon 2023-06-22T18:07:01.755 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-06-22T18:07:01.756 INFO:tasks.ceph.osd.1:Started 2023-06-22T18:07:01.757 INFO:tasks.ceph.osd.2:Restarting daemon 2023-06-22T18:07:01.757 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-06-22T18:07:01.759 INFO:tasks.ceph.osd.2:Started 2023-06-22T18:07:01.759 INFO:tasks.ceph.osd.3:Restarting daemon 2023-06-22T18:07:01.759 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-06-22T18:07:01.761 INFO:tasks.ceph.osd.3:Started 2023-06-22T18:07:01.762 INFO:tasks.ceph.osd.4:Restarting daemon 2023-06-22T18:07:01.762 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-06-22T18:07:01.764 INFO:tasks.ceph.osd.4:Started 2023-06-22T18:07:01.764 INFO:tasks.ceph.osd.5:Restarting daemon 2023-06-22T18:07:01.764 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-06-22T18:07:01.766 INFO:tasks.ceph.osd.5:Started 2023-06-22T18:07:01.766 INFO:tasks.ceph.osd.6:Restarting daemon 2023-06-22T18:07:01.767 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-06-22T18:07:01.768 INFO:tasks.ceph.osd.6:Started 2023-06-22T18:07:01.769 INFO:tasks.ceph.osd.7:Restarting daemon 2023-06-22T18:07:01.769 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-06-22T18:07:01.771 INFO:tasks.ceph.osd.7:Started 2023-06-22T18:07:01.771 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T18:07:01.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.922+0000 7fc0dc923700 1 Processor -- start 2023-06-22T18:07:01.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.922+0000 7fc0dc923700 1 -- start start 2023-06-22T18:07:01.927 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0dc923700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc0d80704c0 0x7fc0d815d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.927 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0dc923700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d815d060 con 0x7fc0d815dd10 2023-06-22T18:07:01.927 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0dc923700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d815d340 con 0x7fc0d80ca890 2023-06-22T18:07:01.927 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0dc923700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d806ecb0 con 0x7fc0d80704c0 2023-06-22T18:07:01.927 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0d6d9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc0d815dd10 0x7fc0d806a0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:41416/0 (socket says 172.21.15.110:41416) 2023-06-22T18:07:01.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0d6d9d700 1 -- 172.21.15.110:0/4263338304 learned_addr learned my addr 172.21.15.110:0/4263338304 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-06-22T18:07:01.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0d5d9b700 1 --2- 172.21.15.110:0/4263338304 >> v2:172.21.15.116:3300/0 conn(0x7fc0d80704c0 0x7fc0d815d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0d559a700 1 -- 172.21.15.110:0/4263338304 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745429684 0 0) 0x7fc0d815d060 con 0x7fc0d815dd10 2023-06-22T18:07:01.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0d559a700 1 -- 172.21.15.110:0/4263338304 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c0003570 con 0x7fc0d815dd10 2023-06-22T18:07:01.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0d5d9b700 1 -- 172.21.15.110:0/4263338304 >> v1:172.21.15.116:6789/0 conn(0x7fc0d80ca890 legacy=0x7fc0d8159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0d5d9b700 1 -- 172.21.15.110:0/4263338304 >> v1:172.21.15.110:6789/0 conn(0x7fc0d815dd10 legacy=0x7fc0d806a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0d5d9b700 1 -- 172.21.15.110:0/4263338304 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d8070960 con 0x7fc0d80704c0 2023-06-22T18:07:01.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0d5d9b700 1 --2- 172.21.15.110:0/4263338304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0d80704c0 0x7fc0d815d7d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0098e0 tx=0x7fc0cc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ec4156efba6c2c4 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:01.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.923+0000 7fc0d559a700 1 -- 172.21.15.110:0/4263338304 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc00b640 con 0x7fc0d80704c0 2023-06-22T18:07:01.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.924+0000 7fc0d559a700 1 -- 172.21.15.110:0/4263338304 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0cc00b7a0 con 0x7fc0d80704c0 2023-06-22T18:07:01.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.924+0000 7fc0d559a700 1 -- 172.21.15.110:0/4263338304 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc00bb40 con 0x7fc0d80704c0 2023-06-22T18:07:01.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.924+0000 7fc0dc923700 1 -- 172.21.15.110:0/4263338304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0d80704c0 msgr2=0x7fc0d815d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.924+0000 7fc0dc923700 1 --2- 172.21.15.110:0/4263338304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0d80704c0 0x7fc0d815d7d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0098e0 tx=0x7fc0cc0091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.924+0000 7fc0dc923700 1 -- 172.21.15.110:0/4263338304 shutdown_connections 2023-06-22T18:07:01.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.924+0000 7fc0dc923700 1 --2- 172.21.15.110:0/4263338304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0d80704c0 0x7fc0d815d7d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.924+0000 7fc0dc923700 1 -- 172.21.15.110:0/4263338304 >> 172.21.15.110:0/4263338304 conn(0x7fc0d80c9d70 msgr2=0x7fc0d8069990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:01.932 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.924+0000 7fc0dc923700 1 -- 172.21.15.110:0/4263338304 shutdown_connections 2023-06-22T18:07:01.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.924+0000 7fc0dc923700 1 -- 172.21.15.110:0/4263338304 wait complete. 2023-06-22T18:07:01.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.925+0000 7fc0dc923700 1 Processor -- start 2023-06-22T18:07:01.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.925+0000 7fc0dc923700 1 -- start start 2023-06-22T18:07:01.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0dc923700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80704c0 0x7fc0d8151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0dc923700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0d80ca890 0x7fc0d8169a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0dc923700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0d815dd10 0x7fc0d80abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:01.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0dc923700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d8159e80 con 0x7fc0d80704c0 2023-06-22T18:07:01.933 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0dc923700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d816a980 con 0x7fc0d80ca890 2023-06-22T18:07:01.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0dc923700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d816a3c0 con 0x7fc0d815dd10 2023-06-22T18:07:01.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0d659c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80704c0 0x7fc0d8151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0d659c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80704c0 0x7fc0d8151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33732/0 (socket says 172.21.15.110:33732) 2023-06-22T18:07:01.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0d659c700 1 -- 172.21.15.110:0/3535170866 learned_addr learned my addr 172.21.15.110:0/3535170866 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:01.934 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0d6d9d700 1 --2- 172.21.15.110:0/3535170866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0d815dd10 0x7fc0d80abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0d5d9b700 1 --2- 172.21.15.110:0/3535170866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0d80ca890 0x7fc0d8169a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:01.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0d659c700 1 -- 172.21.15.110:0/3535170866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0d815dd10 msgr2=0x7fc0d80abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.926+0000 7fc0d659c700 1 --2- 172.21.15.110:0/3535170866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0d815dd10 0x7fc0d80abfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.927+0000 7fc0d659c700 1 -- 172.21.15.110:0/3535170866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0d80ca890 msgr2=0x7fc0d8169a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:01.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.927+0000 7fc0d659c700 1 --2- 172.21.15.110:0/3535170866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0d80ca890 0x7fc0d8169a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:01.935 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.927+0000 7fc0d659c700 1 -- 172.21.15.110:0/3535170866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc009420 con 0x7fc0d80704c0 2023-06-22T18:07:01.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.927+0000 7fc0d6d9d700 1 --2- 172.21.15.110:0/3535170866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0d815dd10 0x7fc0d80abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:01.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.927+0000 7fc0d659c700 1 --2- 172.21.15.110:0/3535170866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80704c0 0x7fc0d8151f60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc0c800c5c0 tx=0x7fc0c800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:01.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.927+0000 7fc0c67fc700 1 -- 172.21.15.110:0/3535170866 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8011590 con 0x7fc0d80704c0 2023-06-22T18:07:01.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.927+0000 7fc0c67fc700 1 -- 172.21.15.110:0/3535170866 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c8011710 con 0x7fc0d80704c0 2023-06-22T18:07:01.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.927+0000 7fc0dc923700 1 -- 172.21.15.110:0/3535170866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d816a6a0 con 0x7fc0d80704c0 2023-06-22T18:07:01.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.927+0000 7fc0c67fc700 1 -- 172.21.15.110:0/3535170866 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8011ad0 con 0x7fc0d80704c0 2023-06-22T18:07:01.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.927+0000 7fc0dc923700 1 -- 172.21.15.110:0/3535170866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d81579f0 con 0x7fc0d80704c0 2023-06-22T18:07:01.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.928+0000 7fc0c67fc700 1 -- 172.21.15.110:0/3535170866 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc0c801c6c0 con 0x7fc0d80704c0 2023-06-22T18:07:01.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.928+0000 7fc0c67fc700 1 -- 172.21.15.110:0/3535170866 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fc0c8012b00 con 0x7fc0d80704c0 2023-06-22T18:07:01.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.928+0000 7fc0dc923700 1 -- 172.21.15.110:0/3535170866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d804c1b0 con 0x7fc0d80704c0 2023-06-22T18:07:01.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.930+0000 7fc0c67fc700 1 -- 172.21.15.110:0/3535170866 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fc0c8017070 con 0x7fc0d80704c0 2023-06-22T18:07:01.966 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:01.962+0000 7fb312f44180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-22T18:07:02.001 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.998+0000 7fc0dc923700 1 -- 172.21.15.110:0/3535170866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0d815a240 con 0x7fc0d80704c0 2023-06-22T18:07:02.002 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:01.999+0000 7fc0c67fc700 1 -- 172.21.15.110:0/3535170866 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fc0c801cb50 con 0x7fc0d80704c0 2023-06-22T18:07:02.002 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:07:02.003 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":10,"fsid":"cbed9da7-33df-4858-8860-5c7cacb062c0","created":"2023-06-22T18:06:59.330561+0000","modified":"2023-06-22T18:07:01.731974+0000","last_up_change":"0.000000","last_in_change":"2023-06-22T18:07:01.731974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66c18197-0807-4f09-9e82-decc44f8bb6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e2e2e368-b287-4ae9-a502-f2d0276de124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89608384-c8bd-4032-98e6-6dd77f29f4de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e20af00e-ab57-4686-87b0-a60c90128569","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9142dbb-15ab-43a0-8501-5ca261a415ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e81f0aa0-d55d-4d72-937e-a1bc82b3117f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"25f64c11-e6e8-46d1-b064-28743c4233df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"608edcd6-65d3-4e76-bbd4-a612a6befa4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T18:07:02.003 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:02.000+0000 7fc0dc923700 1 -- 172.21.15.110:0/3535170866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80704c0 msgr2=0x7fc0d8151f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:02.004 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:02.000+0000 7fc0dc923700 1 --2- 172.21.15.110:0/3535170866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80704c0 0x7fc0d8151f60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc0c800c5c0 tx=0x7fc0c800c8d0 comp rx=0 tx=0).stop 2023-06-22T18:07:02.004 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:02.000+0000 7fc0dc923700 1 -- 172.21.15.110:0/3535170866 shutdown_connections 2023-06-22T18:07:02.004 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:02.000+0000 7fc0dc923700 1 --2- 172.21.15.110:0/3535170866 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc0d815dd10 0x7fc0d80abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:02.004 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:02.000+0000 7fc0dc923700 1 --2- 172.21.15.110:0/3535170866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80704c0 0x7fc0d8151f60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:02.004 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:02.000+0000 7fc0dc923700 1 --2- 172.21.15.110:0/3535170866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0d80ca890 0x7fc0d8169a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:02.004 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:02.000+0000 7fc0dc923700 1 -- 172.21.15.110:0/3535170866 >> 172.21.15.110:0/3535170866 conn(0x7fc0d80c9d70 msgr2=0x7fc0d80c8680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:02.004 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:02.000+0000 7fc0dc923700 1 -- 172.21.15.110:0/3535170866 shutdown_connections 2023-06-22T18:07:02.005 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:02.000+0000 7fc0dc923700 1 -- 172.21.15.110:0/3535170866 wait complete. 2023-06-22T18:07:02.014 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-06-22T18:07:02.015 INFO:tasks.ceph:Waiting for OSDs to come up 2023-06-22T18:07:02.046 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:02.042+0000 7fb312f44180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-22T18:07:02.058 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:02.055+0000 7f0696b21180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-22T18:07:02.123 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:02.120+0000 7fb312f44180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-22T18:07:02.136 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:02.133+0000 7f0696b21180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-22T18:07:02.212 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:02.209+0000 7f0696b21180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-22T18:07:02.357 INFO:tasks.ceph.osd.4.smithi116.stderr:2023-06-22T18:07:02.354+0000 7f600d7cd540 -1 Falling back to public interface 2023-06-22T18:07:02.359 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-06-22T18:07:02.356+0000 7f41e0af0540 -1 Falling back to public interface 2023-06-22T18:07:02.360 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-06-22T18:07:02.356+0000 7f8f2e622540 -1 Falling back to public interface 2023-06-22T18:07:02.360 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-06-22T18:07:02.356+0000 7f8f28a4b700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2023-06-22T18:07:02.365 INFO:tasks.ceph.osd.6.smithi116.stderr:2023-06-22T18:07:02.362+0000 7f2ddfa4a540 -1 Falling back to public interface 2023-06-22T18:07:02.365 INFO:tasks.ceph.osd.7.smithi116.stderr:2023-06-22T18:07:02.362+0000 7ff77c972540 -1 Falling back to public interface 2023-06-22T18:07:02.383 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-06-22T18:07:02.380+0000 7f465d854540 -1 Falling back to public interface 2023-06-22T18:07:02.384 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-06-22T18:07:02.380+0000 7f3eea213540 -1 Falling back to public interface 2023-06-22T18:07:02.384 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-06-22T18:07:02.381+0000 7f4657c7d700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2023-06-22T18:07:02.386 INFO:tasks.ceph.osd.5.smithi116.stderr:2023-06-22T18:07:02.383+0000 7f2c60b49540 -1 Falling back to public interface 2023-06-22T18:07:02.721 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:02.717+0000 7fb312f44180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-22T18:07:02.803 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:02.799+0000 7f0696b21180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-22T18:07:02.820 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-22T18:07:02.859 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:02.856+0000 7fb312f44180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-22T18:07:02.931 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.927+0000 7fdb756f8700 1 Processor -- start 2023-06-22T18:07:02.933 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.927+0000 7fdb756f8700 1 -- start start 2023-06-22T18:07:02.933 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.928+0000 7fdb756f8700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb7006e980 0x7fdb700b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:02.933 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.928+0000 7fdb756f8700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb7006c040 0x7fdb70151340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:02.934 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.928+0000 7fdb756f8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdb7006ed50 0x7fdb70151880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:02.934 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.928+0000 7fdb756f8700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb70151dc0 con 0x7fdb7006ed50 2023-06-22T18:07:02.934 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.928+0000 7fdb756f8700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb70157870 con 0x7fdb7006c040 2023-06-22T18:07:02.934 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.928+0000 7fdb756f8700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb70157a80 con 0x7fdb7006e980 2023-06-22T18:07:02.934 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.928+0000 7fdb6f7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdb7006ed50 0x7fdb70151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:02.934 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.928+0000 7fdb6f7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdb7006ed50 0x7fdb70151880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33736/0 (socket says 172.21.15.110:33736) 2023-06-22T18:07:02.934 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.928+0000 7fdb6f7fe700 1 -- 172.21.15.110:0/1498008538 learned_addr learned my addr 172.21.15.110:0/1498008538 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:02.935 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.928+0000 7fdb6e7fc700 1 --2- 172.21.15.110:0/1498008538 >> v2:172.21.15.116:3300/0 conn(0x7fdb7006c040 0x7fdb70151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:02.935 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.928+0000 7fdb6effd700 1 --2- 172.21.15.110:0/1498008538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb7006e980 0x7fdb700b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:02.935 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.929+0000 7fdb6f7fe700 1 -- 172.21.15.110:0/1498008538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb7006e980 msgr2=0x7fdb700b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:02.935 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.929+0000 7fdb6f7fe700 1 --2- 172.21.15.110:0/1498008538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb7006e980 0x7fdb700b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:02.935 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.929+0000 7fdb6f7fe700 1 -- 172.21.15.110:0/1498008538 >> v2:172.21.15.116:3300/0 conn(0x7fdb7006c040 msgr2=0x7fdb70151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:02.935 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.929+0000 7fdb6f7fe700 1 --2- 172.21.15.110:0/1498008538 >> v2:172.21.15.116:3300/0 conn(0x7fdb7006c040 0x7fdb70151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:02.935 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.929+0000 7fdb6f7fe700 1 -- 172.21.15.110:0/1498008538 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb70157d60 con 0x7fdb7006ed50 2023-06-22T18:07:02.936 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.929+0000 7fdb6f7fe700 1 --2- 172.21.15.110:0/1498008538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb7006ed50 0x7fdb70151880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdb70069850 tx=0x7fdb68009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9eec206cc8ba35 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:02.936 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.929+0000 7fdb6dffb700 1 -- 172.21.15.110:0/1498008538 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6800e070 con 0x7fdb7006ed50 2023-06-22T18:07:02.936 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.929+0000 7fdb6dffb700 1 -- 172.21.15.110:0/1498008538 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6800b970 con 0x7fdb7006ed50 2023-06-22T18:07:02.936 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.929+0000 7fdb6dffb700 1 -- 172.21.15.110:0/1498008538 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb680092e0 con 0x7fdb7006ed50 2023-06-22T18:07:02.936 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.929+0000 7fdb756f8700 1 -- 172.21.15.110:0/1498008538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb7006ed50 msgr2=0x7fdb70151880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:02.936 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.929+0000 7fdb756f8700 1 --2- 172.21.15.110:0/1498008538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb7006ed50 0x7fdb70151880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdb70069850 tx=0x7fdb68009840 comp rx=0 tx=0).stop 2023-06-22T18:07:02.937 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.930+0000 7fdb756f8700 1 -- 172.21.15.110:0/1498008538 shutdown_connections 2023-06-22T18:07:02.937 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.930+0000 7fdb756f8700 1 --2- 172.21.15.110:0/1498008538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb7006ed50 0x7fdb70151880 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:02.937 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.930+0000 7fdb756f8700 1 --2- 172.21.15.110:0/1498008538 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb7006e980 0x7fdb700b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:02.937 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.930+0000 7fdb756f8700 1 --2- 172.21.15.110:0/1498008538 >> v2:172.21.15.116:3300/0 conn(0x7fdb7006c040 0x7fdb70151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:02.937 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.930+0000 7fdb756f8700 1 -- 172.21.15.110:0/1498008538 >> 172.21.15.110:0/1498008538 conn(0x7fdb700cc880 msgr2=0x7fdb700ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:02.937 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.930+0000 7fdb756f8700 1 -- 172.21.15.110:0/1498008538 shutdown_connections 2023-06-22T18:07:02.937 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.930+0000 7fdb756f8700 1 -- 172.21.15.110:0/1498008538 wait complete. 2023-06-22T18:07:02.939 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.931+0000 7fdb756f8700 1 Processor -- start 2023-06-22T18:07:02.940 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.931+0000 7fdb756f8700 1 -- start start 2023-06-22T18:07:02.940 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb756f8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb7006c040 0x7fdb700c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:02.940 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb756f8700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb7006e980 0x7fdb7014a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:02.940 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb756f8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb7006ed50 0x7fdb7006c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:02.940 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb756f8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700c5d30 con 0x7fdb7006c040 2023-06-22T18:07:02.940 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb756f8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb700c2670 con 0x7fdb7006ed50 2023-06-22T18:07:02.941 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb756f8700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7014ab60 con 0x7fdb7006e980 2023-06-22T18:07:02.941 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb6effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb7006c040 0x7fdb700c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:02.941 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb6effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb7006c040 0x7fdb700c6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:33748/0 (socket says 172.21.15.110:33748) 2023-06-22T18:07:02.941 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb6effd700 1 -- 172.21.15.110:0/3687580376 learned_addr learned my addr 172.21.15.110:0/3687580376 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:02.941 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb6e7fc700 1 --2- 172.21.15.110:0/3687580376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb7006e980 0x7fdb7014a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:02.941 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb6f7fe700 1 --2- 172.21.15.110:0/3687580376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb7006ed50 0x7fdb7006c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:02.941 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb6effd700 1 -- 172.21.15.110:0/3687580376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb7006e980 msgr2=0x7fdb7014a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:02.941 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb6effd700 1 --2- 172.21.15.110:0/3687580376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb7006e980 0x7fdb7014a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:02.942 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb6effd700 1 -- 172.21.15.110:0/3687580376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb7006ed50 msgr2=0x7fdb7006c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:02.942 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb6effd700 1 --2- 172.21.15.110:0/3687580376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb7006ed50 0x7fdb7006c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:02.942 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb6effd700 1 -- 172.21.15.110:0/3687580376 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb68009ae0 con 0x7fdb7006c040 2023-06-22T18:07:02.942 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb6effd700 1 --2- 172.21.15.110:0/3687580376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb7006c040 0x7fdb700c6220 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdb6400ecb0 tx=0x7fdb640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:02.942 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb56ffd700 1 -- 172.21.15.110:0/3687580376 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6400c850 con 0x7fdb7006c040 2023-06-22T18:07:02.942 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb56ffd700 1 -- 172.21.15.110:0/3687580376 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6400c9d0 con 0x7fdb7006c040 2023-06-22T18:07:02.943 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb756f8700 1 -- 172.21.15.110:0/3687580376 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb700c18b0 con 0x7fdb7006c040 2023-06-22T18:07:02.943 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb56ffd700 1 -- 172.21.15.110:0/3687580376 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6400cd90 con 0x7fdb7006c040 2023-06-22T18:07:02.943 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb756f8700 1 -- 172.21.15.110:0/3687580376 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb700c1e70 con 0x7fdb7006c040 2023-06-22T18:07:02.943 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb56ffd700 1 -- 172.21.15.110:0/3687580376 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fdb64005c80 con 0x7fdb7006c040 2023-06-22T18:07:02.943 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.932+0000 7fdb56ffd700 1 -- 172.21.15.110:0/3687580376 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fdb64006700 con 0x7fdb7006c040 2023-06-22T18:07:02.943 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.933+0000 7fdb756f8700 1 -- 172.21.15.110:0/3687580376 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb7004c1b0 con 0x7fdb7006c040 2023-06-22T18:07:02.943 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:02.935+0000 7fdb56ffd700 1 -- 172.21.15.110:0/3687580376 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fdb64021060 con 0x7fdb7006c040 2023-06-22T18:07:02.944 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:02.936+0000 7f0696b21180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-22T18:07:02.951 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:02.948+0000 7fb312f44180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-22T18:07:03.008 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:03.004+0000 7fdb756f8700 1 -- 172.21.15.110:0/3687580376 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb70158040 con 0x7fdb7006c040 2023-06-22T18:07:03.008 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:03.005+0000 7fdb56ffd700 1 -- 172.21.15.110:0/3687580376 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fdb64017070 con 0x7fdb7006c040 2023-06-22T18:07:03.009 INFO:teuthology.misc.health.smithi110.stdout: 2023-06-22T18:07:03.009 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":10,"fsid":"cbed9da7-33df-4858-8860-5c7cacb062c0","created":"2023-06-22T18:06:59.330561+0000","modified":"2023-06-22T18:07:01.731974+0000","last_up_change":"0.000000","last_in_change":"2023-06-22T18:07:01.731974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66c18197-0807-4f09-9e82-decc44f8bb6c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e2e2e368-b287-4ae9-a502-f2d0276de124","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"89608384-c8bd-4032-98e6-6dd77f29f4de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e20af00e-ab57-4686-87b0-a60c90128569","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a9142dbb-15ab-43a0-8501-5ca261a415ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e81f0aa0-d55d-4d72-937e-a1bc82b3117f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"25f64c11-e6e8-46d1-b064-28743c4233df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"608edcd6-65d3-4e76-bbd4-a612a6befa4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T18:07:03.010 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:03.006+0000 7fdb756f8700 1 -- 172.21.15.110:0/3687580376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb7006c040 msgr2=0x7fdb700c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:03.010 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:03.006+0000 7fdb756f8700 1 --2- 172.21.15.110:0/3687580376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb7006c040 0x7fdb700c6220 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdb6400ecb0 tx=0x7fdb640091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:03.010 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:03.006+0000 7fdb756f8700 1 -- 172.21.15.110:0/3687580376 shutdown_connections 2023-06-22T18:07:03.011 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:03.006+0000 7fdb756f8700 1 --2- 172.21.15.110:0/3687580376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb7006ed50 0x7fdb7006c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:03.011 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:03.006+0000 7fdb756f8700 1 --2- 172.21.15.110:0/3687580376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb7006c040 0x7fdb700c6220 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:03.011 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:03.006+0000 7fdb756f8700 1 --2- 172.21.15.110:0/3687580376 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fdb7006e980 0x7fdb7014a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:03.011 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:03.006+0000 7fdb756f8700 1 -- 172.21.15.110:0/3687580376 >> 172.21.15.110:0/3687580376 conn(0x7fdb700cc880 msgr2=0x7fdb7014cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:03.011 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:03.006+0000 7fdb756f8700 1 -- 172.21.15.110:0/3687580376 shutdown_connections 2023-06-22T18:07:03.011 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:03.006+0000 7fdb756f8700 1 -- 172.21.15.110:0/3687580376 wait complete. 2023-06-22T18:07:03.021 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-06-22T18:07:03.028 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:03.025+0000 7f0696b21180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-22T18:07:03.201 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:03.198+0000 7fb312f44180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-22T18:07:03.273 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:03.270+0000 7f0696b21180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-22T18:07:03.280 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:03.276+0000 7fb312f44180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-22T18:07:03.350 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:03.347+0000 7f0696b21180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-22T18:07:04.100 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:04.097+0000 7fb312f44180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-22T18:07:04.171 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:04.168+0000 7f0696b21180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-22T18:07:04.172 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:04.169+0000 7fb312f44180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-22T18:07:04.243 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:04.240+0000 7f0696b21180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-22T18:07:04.246 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:04.243+0000 7fb312f44180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-22T18:07:04.318 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:04.315+0000 7f0696b21180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-22T18:07:04.380 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:04.377+0000 7fb312f44180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-22T18:07:04.450 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:04.447+0000 7f0696b21180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-22T18:07:04.463 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:04.460+0000 7fb312f44180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-22T18:07:04.534 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:04.531+0000 7f0696b21180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-22T18:07:04.612 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:04.609+0000 7fb312f44180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-22T18:07:04.688 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:04.684+0000 7fb312f44180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-22T18:07:04.690 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:04.687+0000 7f0696b21180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-22T18:07:04.762 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:04.758+0000 7fb312f44180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-22T18:07:04.767 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:04.764+0000 7f0696b21180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-22T18:07:04.841 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:04.838+0000 7fb312f44180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-22T18:07:04.842 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:04.840+0000 7f0696b21180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-22T18:07:04.959 INFO:tasks.ceph.mgr.x.smithi116.stderr:2023-06-22T18:07:04.956+0000 7f0696b21180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-22T18:07:05.616 INFO:tasks.ceph.osd.7.smithi116.stderr:2023-06-22T18:07:05.613+0000 7ff77c972540 -1 osd.7 0 log_to_monitors true 2023-06-22T18:07:05.647 INFO:tasks.ceph.osd.4.smithi116.stderr:2023-06-22T18:07:05.644+0000 7f600d7cd540 -1 osd.4 0 log_to_monitors true 2023-06-22T18:07:05.659 INFO:tasks.ceph.osd.6.smithi116.stderr:2023-06-22T18:07:05.657+0000 7f2ddfa4a540 -1 osd.6 0 log_to_monitors true 2023-06-22T18:07:05.853 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-06-22T18:07:05.850+0000 7f41e0af0540 -1 osd.0 0 log_to_monitors true 2023-06-22T18:07:05.864 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:05.861+0000 7fb2cc345700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-06-22T18:07:05.865 INFO:tasks.ceph.mgr.y.smithi110.stderr:2023-06-22T18:07:05.861+0000 7fb2cc345700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-06-22T18:07:05.899 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-06-22T18:07:05.895+0000 7f8f2e622540 -1 osd.1 0 log_to_monitors true 2023-06-22T18:07:05.914 INFO:tasks.ceph.osd.5.smithi116.stderr:2023-06-22T18:07:05.911+0000 7f2c60b49540 -1 osd.5 0 log_to_monitors true 2023-06-22T18:07:05.925 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-06-22T18:07:05.921+0000 7f3eea213540 -1 osd.3 0 log_to_monitors true 2023-06-22T18:07:05.927 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-06-22T18:07:05.924+0000 7f465d854540 -1 osd.2 0 log_to_monitors true 2023-06-22T18:07:08.439 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-06-22T18:07:08.436+0000 7f41d7712700 -1 osd.0 0 waiting for initial osdmap 2023-06-22T18:07:08.441 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-06-22T18:07:08.437+0000 7f3ee0e35700 -1 osd.3 0 waiting for initial osdmap 2023-06-22T18:07:08.442 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-06-22T18:07:08.439+0000 7f4654476700 -1 osd.2 0 waiting for initial osdmap 2023-06-22T18:07:08.442 INFO:tasks.ceph.osd.0.smithi110.stderr:2023-06-22T18:07:08.439+0000 7f41d0509700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T18:07:08.446 INFO:tasks.ceph.osd.2.smithi110.stderr:2023-06-22T18:07:08.443+0000 7f464d26d700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T18:07:08.454 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-06-22T18:07:08.451+0000 7f8f26a47700 -1 osd.1 0 waiting for initial osdmap 2023-06-22T18:07:08.457 INFO:tasks.ceph.osd.1.smithi110.stderr:2023-06-22T18:07:08.454+0000 7f8f1e03b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T18:07:08.486 INFO:tasks.ceph.osd.3.smithi110.stderr:2023-06-22T18:07:08.483+0000 7f3ed9c2c700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T18:07:09.133 INFO:tasks.ceph.osd.5.smithi116.stderr:2023-06-22T18:07:09.129+0000 7f2c5776b700 -1 osd.5 0 waiting for initial osdmap 2023-06-22T18:07:09.137 INFO:tasks.ceph.osd.5.smithi116.stderr:2023-06-22T18:07:09.135+0000 7f2c50562700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T18:07:09.143 INFO:tasks.ceph.osd.7.smithi116.stderr:2023-06-22T18:07:09.140+0000 7ff773594700 -1 osd.7 0 waiting for initial osdmap 2023-06-22T18:07:09.149 INFO:tasks.ceph.osd.7.smithi116.stderr:2023-06-22T18:07:09.146+0000 7ff76c38b700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T18:07:09.164 INFO:tasks.ceph.osd.4.smithi116.stderr:2023-06-22T18:07:09.161+0000 7f60043ef700 -1 osd.4 0 waiting for initial osdmap 2023-06-22T18:07:09.169 INFO:tasks.ceph.osd.6.smithi116.stderr:2023-06-22T18:07:09.167+0000 7f2dd666c700 -1 osd.6 0 waiting for initial osdmap 2023-06-22T18:07:09.170 INFO:tasks.ceph.osd.4.smithi116.stderr:2023-06-22T18:07:09.167+0000 7f5ffd1e6700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T18:07:09.173 INFO:tasks.ceph.osd.6.smithi116.stderr:2023-06-22T18:07:09.170+0000 7f2dcf463700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-22T18:07:09.827 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-22T18:07:09.946 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.942+0000 7fe7e8ee7700 1 Processor -- start 2023-06-22T18:07:09.946 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.942+0000 7fe7e8ee7700 1 -- start start 2023-06-22T18:07:09.948 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.943+0000 7fe7e8ee7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe7e41599a0 0x7fe7e415a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:09.948 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.943+0000 7fe7e8ee7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe7e40abb00 0x7fe7e40aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:09.948 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.943+0000 7fe7e8ee7700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e40aff70 con 0x7fe7e40b0400 2023-06-22T18:07:09.948 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.943+0000 7fe7e8ee7700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e4159d70 con 0x7fe7e40abb00 2023-06-22T18:07:09.949 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.943+0000 7fe7e8ee7700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e40af360 con 0x7fe7e41599a0 2023-06-22T18:07:09.949 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.943+0000 7fe7d9d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe7e40abb00 0x7fe7e40aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:09.949 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.943+0000 7fe7d9d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe7e40abb00 0x7fe7e40aaf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42280/0 (socket says 172.21.15.110:42280) 2023-06-22T18:07:09.949 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.943+0000 7fe7d9d9b700 1 -- 172.21.15.110:0/3689042791 learned_addr learned my addr 172.21.15.110:0/3689042791 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:09.949 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.943+0000 7fe7e259c700 1 --2- 172.21.15.110:0/3689042791 >> v2:172.21.15.116:3300/0 conn(0x7fe7e41599a0 0x7fe7e415a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:09.949 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7d9d9b700 1 -- 172.21.15.110:0/3689042791 >> v2:172.21.15.116:3300/0 conn(0x7fe7e41599a0 msgr2=0x7fe7e415a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:09.949 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7d9d9b700 1 --2- 172.21.15.110:0/3689042791 >> v2:172.21.15.116:3300/0 conn(0x7fe7e41599a0 0x7fe7e415a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:09.950 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7d9d9b700 1 -- 172.21.15.110:0/3689042791 >> v1:172.21.15.116:6789/0 conn(0x7fe7e40b0400 legacy=0x7fe7e40ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:09.950 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7d9d9b700 1 -- 172.21.15.110:0/3689042791 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e40af640 con 0x7fe7e40abb00 2023-06-22T18:07:09.950 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7e259c700 1 --2- 172.21.15.110:0/3689042791 >> v2:172.21.15.116:3300/0 conn(0x7fe7e41599a0 0x7fe7e415a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:09.950 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7d9d9b700 1 --2- 172.21.15.110:0/3689042791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7e40abb00 0x7fe7e40aaf30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe7e406a930 tx=0x7fe7cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f1a5eed7c5a555 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:09.950 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7e1d9b700 1 -- 172.21.15.110:0/3689042791 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc009e70 con 0x7fe7e40abb00 2023-06-22T18:07:09.950 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7e1d9b700 1 -- 172.21.15.110:0/3689042791 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc00b860 con 0x7fe7e40abb00 2023-06-22T18:07:09.950 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7e1d9b700 1 -- 172.21.15.110:0/3689042791 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc00bc00 con 0x7fe7e40abb00 2023-06-22T18:07:09.950 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3689042791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7e40abb00 msgr2=0x7fe7e40aaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:09.951 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7e8ee7700 1 --2- 172.21.15.110:0/3689042791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7e40abb00 0x7fe7e40aaf30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe7e406a930 tx=0x7fe7cc009250 comp rx=0 tx=0).stop 2023-06-22T18:07:09.951 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3689042791 shutdown_connections 2023-06-22T18:07:09.951 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7e8ee7700 1 --2- 172.21.15.110:0/3689042791 >> v2:172.21.15.116:3300/0 conn(0x7fe7e41599a0 0x7fe7e415a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:09.951 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.944+0000 7fe7e8ee7700 1 --2- 172.21.15.110:0/3689042791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7e40abb00 0x7fe7e40aaf30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:09.951 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.945+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3689042791 >> 172.21.15.110:0/3689042791 conn(0x7fe7e4097840 msgr2=0x7fe7e4070a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:09.951 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.945+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3689042791 shutdown_connections 2023-06-22T18:07:09.951 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.945+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3689042791 wait complete. 2023-06-22T18:07:09.954 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.946+0000 7fe7e8ee7700 1 Processor -- start 2023-06-22T18:07:09.954 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.946+0000 7fe7e8ee7700 1 -- start start 2023-06-22T18:07:09.954 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7e8ee7700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe7e40abb00 0x7fe7e414bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:09.954 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7e8ee7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7e40b0400 0x7fe7e40b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:09.954 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7e8ee7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7e41599a0 0x7fe7e40b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:09.954 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7e8ee7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e409a0f0 con 0x7fe7e40b0400 2023-06-22T18:07:09.955 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7e8ee7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40994e0 con 0x7fe7e41599a0 2023-06-22T18:07:09.955 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7e8ee7700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40997c0 con 0x7fe7e40abb00 2023-06-22T18:07:09.955 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7d9d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7e40b0400 0x7fe7e40b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:09.955 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7d9d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7e40b0400 0x7fe7e40b2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42284/0 (socket says 172.21.15.110:42284) 2023-06-22T18:07:09.955 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7d9d9b700 1 -- 172.21.15.110:0/3145719513 learned_addr learned my addr 172.21.15.110:0/3145719513 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:09.955 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7e2d9d700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7e41599a0 0x7fe7e40b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:09.955 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7e259c700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe7e40abb00 0x7fe7e414bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:09.956 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7d9d9b700 1 -- 172.21.15.110:0/3145719513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe7e40abb00 msgr2=0x7fe7e414bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:09.956 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7d9d9b700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe7e40abb00 0x7fe7e414bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:09.956 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7d9d9b700 1 -- 172.21.15.110:0/3145719513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7e41599a0 msgr2=0x7fe7e40b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:09.956 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7d9d9b700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7e41599a0 0x7fe7e40b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:09.956 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7d9d9b700 1 -- 172.21.15.110:0/3145719513 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d40093c0 con 0x7fe7e40b0400 2023-06-22T18:07:09.956 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7d9d9b700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7e40b0400 0x7fe7e40b2d90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe7e406b020 tx=0x7fe7cc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:09.956 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7daffd700 1 -- 172.21.15.110:0/3145719513 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc011420 con 0x7fe7e40b0400 2023-06-22T18:07:09.957 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7daffd700 1 -- 172.21.15.110:0/3145719513 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc011580 con 0x7fe7e40b0400 2023-06-22T18:07:09.957 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3145719513 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7cc0094f0 con 0x7fe7e40b0400 2023-06-22T18:07:09.957 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7daffd700 1 -- 172.21.15.110:0/3145719513 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc011940 con 0x7fe7e40b0400 2023-06-22T18:07:09.957 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.947+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3145719513 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e4099d80 con 0x7fe7e40b0400 2023-06-22T18:07:09.957 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.948+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3145719513 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e4064320 con 0x7fe7e40b0400 2023-06-22T18:07:09.957 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.949+0000 7fe7daffd700 1 -- 172.21.15.110:0/3145719513 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7fe7cc02c3b0 con 0x7fe7e40b0400 2023-06-22T18:07:09.957 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.949+0000 7fe7daffd700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe7c4066e00 0x7fe7c40692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:09.957 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.949+0000 7fe7e259c700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe7c4066e00 0x7fe7c40692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:09.958 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.949+0000 7fe7daffd700 1 -- 172.21.15.110:0/3145719513 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7fe7cc085380 con 0x7fe7e40b0400 2023-06-22T18:07:09.958 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.950+0000 7fe7e259c700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe7c4066e00 0x7fe7c40692b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4006ea0 tx=0x7fe7d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:09.958 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:09.954+0000 7fe7daffd700 1 -- 172.21.15.110:0/3145719513 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fe7cc052c60 con 0x7fe7e40b0400 2023-06-22T18:07:10.145 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.142+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3145719513 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7e404c1b0 con 0x7fe7e40b0400 2023-06-22T18:07:10.146 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.143+0000 7fe7daffd700 1 -- 172.21.15.110:0/3145719513 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10540 (secure 0 0 0) 0x7fe7cc056280 con 0x7fe7e40b0400 2023-06-22T18:07:10.146 INFO:teuthology.misc.health.smithi110.stdout: 2023-06-22T18:07:10.146 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":13,"fsid":"cbed9da7-33df-4858-8860-5c7cacb062c0","created":"2023-06-22T18:06:59.330561+0000","modified":"2023-06-22T18:07:09.187638+0000","last_up_change":"2023-06-22T18:07:09.187638+0000","last_in_change":"2023-06-22T18:07:01.731974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66c18197-0807-4f09-9e82-decc44f8bb6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":117902},{"type":"v1","addr":"172.21.15.110:6801","nonce":117902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":117902},{"type":"v1","addr":"172.21.15.110:6805","nonce":117902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":117902},{"type":"v1","addr":"172.21.15.110:6813","nonce":117902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":117902},{"type":"v1","addr":"172.21.15.110:6809","nonce":117902}]},"public_addr":"172.21.15.110:6801/117902","cluster_addr":"172.21.15.110:6805/117902","heartbeat_back_addr":"172.21.15.110:6813/117902","heartbeat_front_addr":"172.21.15.110:6809/117902","state":["exists","up"]},{"osd":1,"uuid":"e2e2e368-b287-4ae9-a502-f2d0276de124","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":117903},{"type":"v1","addr":"172.21.15.110:6804","nonce":117903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":117903},{"type":"v1","addr":"172.21.15.110:6808","nonce":117903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":117903},{"type":"v1","addr":"172.21.15.110:6815","nonce":117903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":117903},{"type":"v1","addr":"172.21.15.110:6812","nonce":117903}]},"public_addr":"172.21.15.110:6804/117903","cluster_addr":"172.21.15.110:6808/117903","heartbeat_back_addr":"172.21.15.110:6815/117903","heartbeat_front_addr":"172.21.15.110:6812/117903","state":["exists","up"]},{"osd":2,"uuid":"89608384-c8bd-4032-98e6-6dd77f29f4de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":117932},{"type":"v1","addr":"172.21.15.110:6819","nonce":117932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6821","nonce":117932},{"type":"v1","addr":"172.21.15.110:6823","nonce":117932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6829","nonce":117932},{"type":"v1","addr":"172.21.15.110:6831","nonce":117932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6825","nonce":117932},{"type":"v1","addr":"172.21.15.110:6827","nonce":117932}]},"public_addr":"172.21.15.110:6819/117932","cluster_addr":"172.21.15.110:6823/117932","heartbeat_back_addr":"172.21.15.110:6831/117932","heartbeat_front_addr":"172.21.15.110:6827/117932","state":["exists","up"]},{"osd":3,"uuid":"e20af00e-ab57-4686-87b0-a60c90128569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":117933},{"type":"v1","addr":"172.21.15.110:6818","nonce":117933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":117933},{"type":"v1","addr":"172.21.15.110:6822","nonce":117933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":117933},{"type":"v1","addr":"172.21.15.110:6830","nonce":117933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":117933},{"type":"v1","addr":"172.21.15.110:6826","nonce":117933}]},"public_addr":"172.21.15.110:6818/117933","cluster_addr":"172.21.15.110:6822/117933","heartbeat_back_addr":"172.21.15.110:6830/117933","heartbeat_front_addr":"172.21.15.110:6826/117933","state":["exists","up"]},{"osd":4,"uuid":"a9142dbb-15ab-43a0-8501-5ca261a415ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":118295},{"type":"v1","addr":"172.21.15.116:6801","nonce":118295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":118295},{"type":"v1","addr":"172.21.15.116:6803","nonce":118295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":118295},{"type":"v1","addr":"172.21.15.116:6807","nonce":118295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":118295},{"type":"v1","addr":"172.21.15.116:6805","nonce":118295}]},"public_addr":"172.21.15.116:6801/118295","cluster_addr":"172.21.15.116:6803/118295","heartbeat_back_addr":"172.21.15.116:6807/118295","heartbeat_front_addr":"172.21.15.116:6805/118295","state":["exists","up"]},{"osd":5,"uuid":"e81f0aa0-d55d-4d72-937e-a1bc82b3117f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":118312},{"type":"v1","addr":"172.21.15.116:6825","nonce":118312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":118312},{"type":"v1","addr":"172.21.15.116:6827","nonce":118312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":118312},{"type":"v1","addr":"172.21.15.116:6831","nonce":118312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":118312},{"type":"v1","addr":"172.21.15.116:6829","nonce":118312}]},"public_addr":"172.21.15.116:6825/118312","cluster_addr":"172.21.15.116:6827/118312","heartbeat_back_addr":"172.21.15.116:6831/118312","heartbeat_front_addr":"172.21.15.116:6829/118312","state":["exists","up"]},{"osd":6,"uuid":"25f64c11-e6e8-46d1-b064-28743c4233df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":118296},{"type":"v1","addr":"172.21.15.116:6810","nonce":118296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":118296},{"type":"v1","addr":"172.21.15.116:6814","nonce":118296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":118296},{"type":"v1","addr":"172.21.15.116:6822","nonce":118296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":118296},{"type":"v1","addr":"172.21.15.116:6818","nonce":118296}]},"public_addr":"172.21.15.116:6810/118296","cluster_addr":"172.21.15.116:6814/118296","heartbeat_back_addr":"172.21.15.116:6822/118296","heartbeat_front_addr":"172.21.15.116:6818/118296","state":["exists","up"]},{"osd":7,"uuid":"608edcd6-65d3-4e76-bbd4-a612a6befa4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":118297},{"type":"v1","addr":"172.21.15.116:6811","nonce":118297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":118297},{"type":"v1","addr":"172.21.15.116:6815","nonce":118297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":118297},{"type":"v1","addr":"172.21.15.116:6823","nonce":118297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":118297},{"type":"v1","addr":"172.21.15.116:6819","nonce":118297}]},"public_addr":"172.21.15.116:6811/118297","cluster_addr":"172.21.15.116:6815/118297","heartbeat_back_addr":"172.21.15.116:6823/118297","heartbeat_front_addr":"172.21.15.116:6819/118297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T18:07:10.148 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3145719513 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe7c4066e00 msgr2=0x7fe7c40692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:10.148 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe7c4066e00 0x7fe7c40692b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4006ea0 tx=0x7fe7d4008040 comp rx=0 tx=0).stop 2023-06-22T18:07:10.148 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3145719513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7e40b0400 msgr2=0x7fe7e40b2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:10.149 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7e40b0400 0x7fe7e40b2d90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe7e406b020 tx=0x7fe7cc017000 comp rx=0 tx=0).stop 2023-06-22T18:07:10.149 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3145719513 shutdown_connections 2023-06-22T18:07:10.149 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7e41599a0 0x7fe7e40b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:10.149 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe7c4066e00 0x7fe7c40692b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:10.149 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe7e40abb00 0x7fe7e414bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:10.149 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 --2- 172.21.15.110:0/3145719513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7e40b0400 0x7fe7e40b2d90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:10.149 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3145719513 >> 172.21.15.110:0/3145719513 conn(0x7fe7e4097840 msgr2=0x7fe7e4158d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:10.150 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3145719513 shutdown_connections 2023-06-22T18:07:10.150 INFO:teuthology.misc.health.smithi110.stderr:2023-06-22T18:07:10.145+0000 7fe7e8ee7700 1 -- 172.21.15.110:0/3145719513 wait complete. 2023-06-22T18:07:10.159 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-06-22T18:07:10.159 INFO:tasks.ceph:Creating RBD pool 2023-06-22T18:07:10.159 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool create rbd 8 2023-06-22T18:07:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.271+0000 7fc57b5fb700 1 Processor -- start 2023-06-22T18:07:10.275 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.272+0000 7fc57b5fb700 1 -- start start 2023-06-22T18:07:10.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.272+0000 7fc57b5fb700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574155950 0x7fc5740ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:10.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.272+0000 7fc57b5fb700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5740aea10 con 0x7fc574155d20 2023-06-22T18:07:10.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.272+0000 7fc57b5fb700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5740c89a0 con 0x7fc5740afc40 2023-06-22T18:07:10.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.272+0000 7fc57b5fb700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5740c8c80 con 0x7fc574155950 2023-06-22T18:07:10.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.272+0000 7fc578b96700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc574155d20 0x7fc5740af530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:42310/0 (socket says 172.21.15.110:42310) 2023-06-22T18:07:10.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc578b96700 1 -- 172.21.15.110:0/3135840025 learned_addr learned my addr 172.21.15.110:0/3135840025 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-06-22T18:07:10.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc579397700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574155950 0x7fc5740ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:10.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003963031 0 0) 0x7fc5740aea10 con 0x7fc574155d20 2023-06-22T18:07:10.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc560003570 con 0x7fc574155d20 2023-06-22T18:07:10.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2581243728 0 0) 0x7fc560003570 con 0x7fc574155d20 2023-06-22T18:07:10.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574155950 msgr2=0x7fc5740ae1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:10.279 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574155950 0x7fc5740ae1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 >> v1:172.21.15.116:6789/0 conn(0x7fc5740afc40 legacy=0x7fc5740c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5740c8f60 con 0x7fc574155d20 2023-06-22T18:07:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 519278953 0 0) 0x7fc564001470 con 0x7fc574155d20 2023-06-22T18:07:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 >> v1:172.21.15.110:6789/0 conn(0x7fc574155d20 legacy=0x7fc5740af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 0x7fc560003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560004810 0x7fc560004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574155950 0x7fc5600051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5740c8c80 con 0x7fc560003a90 2023-06-22T18:07:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56000bb60 con 0x7fc560004810 2023-06-22T18:07:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56000bd20 con 0x7fc574155950 2023-06-22T18:07:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc579397700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 0x7fc560003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc578b96700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560004810 0x7fc560004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc579b98700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574155950 0x7fc5600051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc579397700 1 -- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574155950 msgr2=0x7fc5600051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc579397700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574155950 0x7fc5600051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc579397700 1 -- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560004810 msgr2=0x7fc560004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc579397700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560004810 0x7fc560004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc579397700 1 -- 172.21.15.110:0/3135840025 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc560003570 con 0x7fc560003a90 2023-06-22T18:07:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.273+0000 7fc578b96700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc560004810 0x7fc560004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T18:07:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.274+0000 7fc579397700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 0x7fc560003ea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc57000f3c0 tx=0x7fc57000c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbe9c3a990e28da server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.274+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57000a0d0 con 0x7fc560003a90 2023-06-22T18:07:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.274+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc570010040 con 0x7fc560003a90 2023-06-22T18:07:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.274+0000 7fc56bfff700 1 -- 172.21.15.110:0/3135840025 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57000fc10 con 0x7fc560003a90 2023-06-22T18:07:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.274+0000 7fc57b5fb700 1 -- 172.21.15.110:0/3135840025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 msgr2=0x7fc560003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.274+0000 7fc57b5fb700 1 --2- 172.21.15.110:0/3135840025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 0x7fc560003ea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc57000f3c0 tx=0x7fc57000c040 comp rx=0 tx=0).stop 2023-06-22T18:07:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.274+0000 7fc579b98700 1 -- 172.21.15.110:0/3135840025 reap_dead start 2023-06-22T18:07:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.274+0000 7fc57b5fb700 1 -- 172.21.15.110:0/3135840025 shutdown_connections 2023-06-22T18:07:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.274+0000 7fc57b5fb700 1 -- 172.21.15.110:0/3135840025 >> 172.21.15.110:0/3135840025 conn(0x7fc5740cbfe0 msgr2=0x7fc5740cc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.274+0000 7fc57b5fb700 1 -- 172.21.15.110:0/3135840025 shutdown_connections 2023-06-22T18:07:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.274+0000 7fc57b5fb700 1 -- 172.21.15.110:0/3135840025 wait complete. 2023-06-22T18:07:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.275+0000 7fc57b5fb700 1 Processor -- start 2023-06-22T18:07:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.275+0000 7fc57b5fb700 1 -- start start 2023-06-22T18:07:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.276+0000 7fc57b5fb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 0x7fc57414c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.276+0000 7fc57b5fb700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574145ef0 0x7fc57406d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.276+0000 7fc57b5fb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc574146360 0x7fc57406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.276+0000 7fc57b5fb700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc574153f10 con 0x7fc560003a90 2023-06-22T18:07:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.276+0000 7fc57b5fb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc574153a80 con 0x7fc574146360 2023-06-22T18:07:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.276+0000 7fc57b5fb700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc57414cd60 con 0x7fc574145ef0 2023-06-22T18:07:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.276+0000 7fc579397700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 0x7fc57414c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.276+0000 7fc579397700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 0x7fc57414c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42300/0 (socket says 172.21.15.110:42300) 2023-06-22T18:07:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.276+0000 7fc579397700 1 -- 172.21.15.110:0/1346507351 learned_addr learned my addr 172.21.15.110:0/1346507351 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.276+0000 7fc579b98700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc574146360 0x7fc57406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.276+0000 7fc578b96700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574145ef0 0x7fc57406d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.277+0000 7fc579397700 1 -- 172.21.15.110:0/1346507351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574145ef0 msgr2=0x7fc57406d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.277+0000 7fc579397700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574145ef0 0x7fc57406d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.277+0000 7fc579397700 1 -- 172.21.15.110:0/1346507351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc574146360 msgr2=0x7fc57406d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.277+0000 7fc579397700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc574146360 0x7fc57406d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.277+0000 7fc579397700 1 -- 172.21.15.110:0/1346507351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc574152e70 con 0x7fc560003a90 2023-06-22T18:07:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.277+0000 7fc579397700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 0x7fc57414c820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc570016000 tx=0x7fc57001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.277+0000 7fc5697fa700 1 -- 172.21.15.110:0/1346507351 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570015550 con 0x7fc560003a90 2023-06-22T18:07:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.277+0000 7fc5697fa700 1 -- 172.21.15.110:0/1346507351 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc570023070 con 0x7fc560003a90 2023-06-22T18:07:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.277+0000 7fc5697fa700 1 -- 172.21.15.110:0/1346507351 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570015940 con 0x7fc560003a90 2023-06-22T18:07:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.277+0000 7fc57b5fb700 1 -- 172.21.15.110:0/1346507351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc574153150 con 0x7fc560003a90 2023-06-22T18:07:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.277+0000 7fc57b5fb700 1 -- 172.21.15.110:0/1346507351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc574153710 con 0x7fc560003a90 2023-06-22T18:07:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.278+0000 7fc57b5fb700 1 -- 172.21.15.110:0/1346507351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5740ae7b0 con 0x7fc560003a90 2023-06-22T18:07:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.278+0000 7fc5697fa700 1 -- 172.21.15.110:0/1346507351 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7fc570025030 con 0x7fc560003a90 2023-06-22T18:07:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.278+0000 7fc5697fa700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fc558066e00 0x7fc5580692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.278+0000 7fc5697fa700 1 -- 172.21.15.110:0/1346507351 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4467+0+0 (secure 0 0 0) 0x7fc5700858b0 con 0x7fc560003a90 2023-06-22T18:07:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.278+0000 7fc578b96700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fc558066e00 0x7fc5580692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.279+0000 7fc578b96700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fc558066e00 0x7fc5580692b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc564000c00 tx=0x7fc564007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.283+0000 7fc5697fa700 1 -- 172.21.15.110:0/1346507351 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fc570053190 con 0x7fc560003a90 2023-06-22T18:07:10.470 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:10.466+0000 7fc57b5fb700 1 -- 172.21.15.110:0/1346507351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fc57404c1b0 con 0x7fc560003a90 2023-06-22T18:07:11.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.203+0000 7fc5697fa700 1 -- 172.21.15.110:0/1346507351 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) v1 ==== 109+0+0 (secure 0 0 0) 0x7fc5700567b0 con 0x7fc560003a90 2023-06-22T18:07:11.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 -- 172.21.15.110:0/1346507351 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fc558066e00 msgr2=0x7fc5580692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:11.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fc558066e00 0x7fc5580692b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc564000c00 tx=0x7fc564007040 comp rx=0 tx=0).stop 2023-06-22T18:07:11.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 -- 172.21.15.110:0/1346507351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 msgr2=0x7fc57414c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:11.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 0x7fc57414c820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc570016000 tx=0x7fc57001a000 comp rx=0 tx=0).stop 2023-06-22T18:07:11.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 -- 172.21.15.110:0/1346507351 shutdown_connections 2023-06-22T18:07:11.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc574146360 0x7fc57406d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:11.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fc558066e00 0x7fc5580692b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:11.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc560003a90 0x7fc57414c820 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:11.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 --2- 172.21.15.110:0/1346507351 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fc574145ef0 0x7fc57406d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:11.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 -- 172.21.15.110:0/1346507351 >> 172.21.15.110:0/1346507351 conn(0x7fc5740cbfe0 msgr2=0x7fc57406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:11.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 -- 172.21.15.110:0/1346507351 shutdown_connections 2023-06-22T18:07:11.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.206+0000 7fc57b5fb700 1 -- 172.21.15.110:0/1346507351 wait complete. 2023-06-22T18:07:11.211 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'rbd' created 2023-06-22T18:07:11.222 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-06-22T18:07:11.340 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.336+0000 7f70c1c8e700 1 Processor -- start 2023-06-22T18:07:11.340 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.336+0000 7f70c1c8e700 1 -- start start 2023-06-22T18:07:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70c1c8e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f70bc15aaa0 0x7f70bc0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70c1c8e700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f70bc15df20 0x7f70bc0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70c1c8e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70bc15a610 con 0x7f70bc142f30 2023-06-22T18:07:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70c1c8e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70bc06bfd0 con 0x7f70bc15aaa0 2023-06-22T18:07:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70c1c8e700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70bc06bcf0 con 0x7f70bc15df20 2023-06-22T18:07:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70baffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f70bc15aaa0 0x7f70bc0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70bb7fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f70bc142f30 0x7f70bc06b510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:42314/0 (socket says 172.21.15.110:42314) 2023-06-22T18:07:11.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70bb7fe700 1 -- 172.21.15.110:0/615880148 learned_addr learned my addr 172.21.15.110:0/615880148 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-06-22T18:07:11.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70bbfff700 1 --2- 172.21.15.110:0/615880148 >> v2:172.21.15.116:3300/0 conn(0x7f70bc15df20 0x7f70bc0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:11.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70baffd700 1 -- 172.21.15.110:0/615880148 >> v2:172.21.15.116:3300/0 conn(0x7f70bc15df20 msgr2=0x7f70bc0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:11.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70baffd700 1 --2- 172.21.15.110:0/615880148 >> v2:172.21.15.116:3300/0 conn(0x7f70bc15df20 0x7f70bc0cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:11.348 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70baffd700 1 -- 172.21.15.110:0/615880148 >> v1:172.21.15.110:6789/0 conn(0x7f70bc142f30 legacy=0x7f70bc06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70ba7fc700 1 -- 172.21.15.110:0/615880148 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791190571 0 0) 0x7f70bc15a610 con 0x7f70bc142f30 2023-06-22T18:07:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70baffd700 1 -- 172.21.15.110:0/615880148 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70bc159fc0 con 0x7f70bc15aaa0 2023-06-22T18:07:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70bbfff700 1 --2- 172.21.15.110:0/615880148 >> v2:172.21.15.116:3300/0 conn(0x7f70bc15df20 0x7f70bc0cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T18:07:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70baffd700 1 --2- 172.21.15.110:0/615880148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70bc15aaa0 0x7f70bc0cbdf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f70bc0cf310 tx=0x7f70b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20209d2f79dd20d6 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70ba7fc700 1 -- 172.21.15.110:0/615880148 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b000b6b0 con 0x7f70bc15aaa0 2023-06-22T18:07:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70ba7fc700 1 -- 172.21.15.110:0/615880148 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b000b810 con 0x7f70bc15aaa0 2023-06-22T18:07:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.337+0000 7f70ba7fc700 1 -- 172.21.15.110:0/615880148 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b000bbb0 con 0x7f70bc15aaa0 2023-06-22T18:07:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.338+0000 7f70c1c8e700 1 -- 172.21.15.110:0/615880148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70bc15aaa0 msgr2=0x7f70bc0cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:11.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.338+0000 7f70c1c8e700 1 --2- 172.21.15.110:0/615880148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70bc15aaa0 0x7f70bc0cbdf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f70bc0cf310 tx=0x7f70b00091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:11.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.338+0000 7f70c1c8e700 1 -- 172.21.15.110:0/615880148 shutdown_connections 2023-06-22T18:07:11.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.338+0000 7f70c1c8e700 1 --2- 172.21.15.110:0/615880148 >> v2:172.21.15.116:3300/0 conn(0x7f70bc15df20 0x7f70bc0cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:11.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.338+0000 7f70c1c8e700 1 --2- 172.21.15.110:0/615880148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70bc15aaa0 0x7f70bc0cbdf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:11.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.338+0000 7f70c1c8e700 1 -- 172.21.15.110:0/615880148 >> 172.21.15.110:0/615880148 conn(0x7f70bc0ce220 msgr2=0x7f70bc0ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.338+0000 7f70c1c8e700 1 -- 172.21.15.110:0/615880148 shutdown_connections 2023-06-22T18:07:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.338+0000 7f70c1c8e700 1 -- 172.21.15.110:0/615880148 wait complete. 2023-06-22T18:07:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.339+0000 7f70c1c8e700 1 Processor -- start 2023-06-22T18:07:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.339+0000 7f70c1c8e700 1 -- start start 2023-06-22T18:07:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70c1c8e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70bc142f30 0x7f70bc0aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70c1c8e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70bc15aaa0 0x7f70bc0b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70c1c8e700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70bc15df20 0x7f70bc0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70c1c8e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70bc0abb40 con 0x7f70bc15aaa0 2023-06-22T18:07:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70c1c8e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70bc0aaf30 con 0x7f70bc142f30 2023-06-22T18:07:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70c1c8e700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70bc0ab210 con 0x7f70bc15df20 2023-06-22T18:07:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70baffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70bc15aaa0 0x7f70bc0b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70baffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70bc15aaa0 0x7f70bc0b0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42312/0 (socket says 172.21.15.110:42312) 2023-06-22T18:07:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70baffd700 1 -- 172.21.15.110:0/181656168 learned_addr learned my addr 172.21.15.110:0/181656168 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70bb7fe700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70bc142f30 0x7f70bc0aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:11.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70bbfff700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70bc15df20 0x7f70bc0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:11.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70baffd700 1 -- 172.21.15.110:0/181656168 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70bc15df20 msgr2=0x7f70bc0b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:11.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70baffd700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70bc15df20 0x7f70bc0b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:11.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70baffd700 1 -- 172.21.15.110:0/181656168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70bc142f30 msgr2=0x7f70bc0aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:11.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70baffd700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70bc142f30 0x7f70bc0aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:11.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70baffd700 1 -- 172.21.15.110:0/181656168 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b0009420 con 0x7f70bc15aaa0 2023-06-22T18:07:11.353 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.340+0000 7f70bb7fe700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70bc142f30 0x7f70bc0aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T18:07:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.341+0000 7f70baffd700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70bc15aaa0 0x7f70bc0b0fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f70bc06ac60 tx=0x7f70b0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.341+0000 7f70a37fe700 1 -- 172.21.15.110:0/181656168 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b0011420 con 0x7f70bc15aaa0 2023-06-22T18:07:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.341+0000 7f70a37fe700 1 -- 172.21.15.110:0/181656168 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b0011580 con 0x7f70bc15aaa0 2023-06-22T18:07:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.341+0000 7f70c1c8e700 1 -- 172.21.15.110:0/181656168 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70bc0ab4f0 con 0x7f70bc15aaa0 2023-06-22T18:07:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.341+0000 7f70a37fe700 1 -- 172.21.15.110:0/181656168 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b0011940 con 0x7f70bc15aaa0 2023-06-22T18:07:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.341+0000 7f70c1c8e700 1 -- 172.21.15.110:0/181656168 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70bc0b0820 con 0x7f70bc15aaa0 2023-06-22T18:07:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.342+0000 7f70c1c8e700 1 -- 172.21.15.110:0/181656168 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70bc0ab990 con 0x7f70bc15aaa0 2023-06-22T18:07:11.355 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.342+0000 7f70a37fe700 1 -- 172.21.15.110:0/181656168 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f70b0011d30 con 0x7f70bc15aaa0 2023-06-22T18:07:11.355 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.343+0000 7f70a37fe700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f70a8066e00 0x7f70a80692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.343+0000 7f70a37fe700 1 -- 172.21.15.110:0/181656168 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5184+0+0 (secure 0 0 0) 0x7f70b00857e0 con 0x7f70bc15aaa0 2023-06-22T18:07:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.343+0000 7f70bb7fe700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f70a8066e00 0x7f70a80692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.343+0000 7f70bb7fe700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f70a8066e00 0x7f70a80692b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f70ac001560 tx=0x7f70ac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.347+0000 7f70a37fe700 1 -- 172.21.15.110:0/181656168 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f70b0052e00 con 0x7f70bc15aaa0 2023-06-22T18:07:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:11.533+0000 7f70c1c8e700 1 -- 172.21.15.110:0/181656168 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f70bc04c1b0 con 0x7f70bc15aaa0 2023-06-22T18:07:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.205+0000 7f70a37fe700 1 -- 172.21.15.110:0/181656168 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v16) v1 ==== 173+0+0 (secure 0 0 0) 0x7f70b0028020 con 0x7f70bc15aaa0 2023-06-22T18:07:12.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 -- 172.21.15.110:0/181656168 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f70a8066e00 msgr2=0x7f70a80692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f70a8066e00 0x7f70a80692b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f70ac001560 tx=0x7f70ac009040 comp rx=0 tx=0).stop 2023-06-22T18:07:12.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 -- 172.21.15.110:0/181656168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70bc15aaa0 msgr2=0x7f70bc0b0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70bc15aaa0 0x7f70bc0b0fc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f70bc06ac60 tx=0x7f70b0017000 comp rx=0 tx=0).stop 2023-06-22T18:07:12.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 -- 172.21.15.110:0/181656168 shutdown_connections 2023-06-22T18:07:12.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70bc142f30 0x7f70bc0aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f70a8066e00 0x7f70a80692b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f70bc15df20 0x7f70bc0b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 --2- 172.21.15.110:0/181656168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70bc15aaa0 0x7f70bc0b0fc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 -- 172.21.15.110:0/181656168 >> 172.21.15.110:0/181656168 conn(0x7f70bc0ce220 msgr2=0x7f70bc0c43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:12.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 -- 172.21.15.110:0/181656168 shutdown_connections 2023-06-22T18:07:12.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.208+0000 7f70c1c8e700 1 -- 172.21.15.110:0/181656168 wait complete. 2023-06-22T18:07:12.213 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rbd' on pool 'rbd' 2023-06-22T18:07:12.226 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-06-22T18:07:12.227 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-22T18:07:12.227 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-22T18:07:12.229 INFO:tasks.ceph.mds.a:Started 2023-06-22T18:07:12.229 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-22T18:07:12.230 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-22T18:07:12.231 INFO:tasks.ceph.mds.c:Started 2023-06-22T18:07:12.232 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-22T18:07:12.232 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-22T18:07:12.234 INFO:tasks.ceph.mds.b:Started 2023-06-22T18:07:12.234 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-22T18:07:12.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-22T18:07:12.236 INFO:tasks.ceph.mds.d:Started 2023-06-22T18:07:12.237 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-06-22T18:07:12.237 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T18:07:12.313 INFO:tasks.ceph.mds.a.smithi110.stdout:starting mds.a at 2023-06-22T18:07:12.315 INFO:tasks.ceph.mds.c.smithi110.stdout:starting mds.c at 2023-06-22T18:07:12.318 INFO:tasks.ceph.mds.b.smithi116.stdout:starting mds.b at 2023-06-22T18:07:12.319 INFO:tasks.ceph.mds.d.smithi116.stdout:starting mds.d at 2023-06-22T18:07:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.368+0000 7f278557d700 1 Processor -- start 2023-06-22T18:07:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.368+0000 7f278557d700 1 -- start start 2023-06-22T18:07:12.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f278557d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2780157c20 0x7f278006d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f278557d700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27800cab00 0x7f27800caf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f278557d700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27800c8e00 con 0x7f27800c8890 2023-06-22T18:07:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f278557d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27800c90e0 con 0x7f2780157c20 2023-06-22T18:07:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f278557d700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278006cbc0 con 0x7f27800cab00 2023-06-22T18:07:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f277effd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2780157c20 0x7f278006d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f277effd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2780157c20 0x7f278006d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42354/0 (socket says 172.21.15.110:42354) 2023-06-22T18:07:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f277effd700 1 -- 172.21.15.110:0/2970701401 learned_addr learned my addr 172.21.15.110:0/2970701401 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f277f7fe700 1 --2- 172.21.15.110:0/2970701401 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27800cab00 0x7f27800caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f277effd700 1 -- 172.21.15.110:0/2970701401 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27800cab00 msgr2=0x7f27800caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f277effd700 1 --2- 172.21.15.110:0/2970701401 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27800cab00 0x7f27800caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f277effd700 1 -- 172.21.15.110:0/2970701401 >> v1:172.21.15.116:6789/0 conn(0x7f27800c8890 legacy=0x7f2780157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f277effd700 1 -- 172.21.15.110:0/2970701401 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278006cea0 con 0x7f2780157c20 2023-06-22T18:07:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.369+0000 7f277effd700 1 --2- 172.21.15.110:0/2970701401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2780157c20 0x7f278006d6a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f27740099b0 tx=0x7f2774009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40da0a6c525d9056 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.370+0000 7f277dffb700 1 -- 172.21.15.110:0/2970701401 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2774009ea0 con 0x7f2780157c20 2023-06-22T18:07:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.370+0000 7f277dffb700 1 -- 172.21.15.110:0/2970701401 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277400b860 con 0x7f2780157c20 2023-06-22T18:07:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.370+0000 7f277dffb700 1 -- 172.21.15.110:0/2970701401 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277400bc00 con 0x7f2780157c20 2023-06-22T18:07:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.370+0000 7f278557d700 1 -- 172.21.15.110:0/2970701401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2780157c20 msgr2=0x7f278006d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.370+0000 7f278557d700 1 --2- 172.21.15.110:0/2970701401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2780157c20 0x7f278006d6a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f27740099b0 tx=0x7f2774009250 comp rx=0 tx=0).stop 2023-06-22T18:07:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.370+0000 7f278557d700 1 -- 172.21.15.110:0/2970701401 shutdown_connections 2023-06-22T18:07:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.370+0000 7f278557d700 1 --2- 172.21.15.110:0/2970701401 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27800cab00 0x7f27800caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.370+0000 7f278557d700 1 --2- 172.21.15.110:0/2970701401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2780157c20 0x7f278006d6a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.370+0000 7f278557d700 1 -- 172.21.15.110:0/2970701401 >> 172.21.15.110:0/2970701401 conn(0x7f278006b550 msgr2=0x7f278006b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.370+0000 7f278557d700 1 -- 172.21.15.110:0/2970701401 shutdown_connections 2023-06-22T18:07:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.370+0000 7f278557d700 1 -- 172.21.15.110:0/2970701401 wait complete. 2023-06-22T18:07:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.371+0000 7f278557d700 1 Processor -- start 2023-06-22T18:07:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f278557d700 1 -- start start 2023-06-22T18:07:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f278557d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27800c8890 0x7f27801507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f278557d700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27800cab00 0x7f2780158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f278557d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2780157c20 0x7f278015ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f278557d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801565d0 con 0x7f2780157c20 2023-06-22T18:07:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f278557d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801685b0 con 0x7f27800c8890 2023-06-22T18:07:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f278557d700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2780155f80 con 0x7f27800cab00 2023-06-22T18:07:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f277f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2780157c20 0x7f278015ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f277effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27800c8890 0x7f27801507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f277f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2780157c20 0x7f278015ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42368/0 (socket says 172.21.15.110:42368) 2023-06-22T18:07:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f277f7fe700 1 -- 172.21.15.110:0/1581902953 learned_addr learned my addr 172.21.15.110:0/1581902953 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.372+0000 7f277e7fc700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27800cab00 0x7f2780158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f277f7fe700 1 -- 172.21.15.110:0/1581902953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27800cab00 msgr2=0x7f2780158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f277f7fe700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27800cab00 0x7f2780158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f277f7fe700 1 -- 172.21.15.110:0/1581902953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27800c8890 msgr2=0x7f27801507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f277f7fe700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27800c8890 0x7f27801507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f277f7fe700 1 -- 172.21.15.110:0/1581902953 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27740094f0 con 0x7f2780157c20 2023-06-22T18:07:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f277effd700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27800c8890 0x7f27801507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f277f7fe700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2780157c20 0x7f278015ae00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2778009df0 tx=0x7f277800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f2766ffd700 1 -- 172.21.15.110:0/1581902953 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277800caa0 con 0x7f2780157c20 2023-06-22T18:07:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f2766ffd700 1 -- 172.21.15.110:0/1581902953 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277800f040 con 0x7f2780157c20 2023-06-22T18:07:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f278557d700 1 -- 172.21.15.110:0/1581902953 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2780156260 con 0x7f2780157c20 2023-06-22T18:07:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f2766ffd700 1 -- 172.21.15.110:0/1581902953 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277800ce70 con 0x7f2780157c20 2023-06-22T18:07:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.373+0000 7f278557d700 1 -- 172.21.15.110:0/1581902953 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2780159070 con 0x7f2780157c20 2023-06-22T18:07:12.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.374+0000 7f278557d700 1 -- 172.21.15.110:0/1581902953 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2780155df0 con 0x7f2780157c20 2023-06-22T18:07:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.374+0000 7f2766ffd700 1 -- 172.21.15.110:0/1581902953 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f2778014690 con 0x7f2780157c20 2023-06-22T18:07:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.375+0000 7f2766ffd700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f276c066e00 0x7f276c0692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.375+0000 7f2766ffd700 1 -- 172.21.15.110:0/1581902953 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2778084920 con 0x7f2780157c20 2023-06-22T18:07:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.375+0000 7f277effd700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f276c066e00 0x7f276c0692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.375+0000 7f277effd700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f276c066e00 0x7f276c0692b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f27740099b0 tx=0x7f2774017000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.383+0000 7f2766ffd700 1 -- 172.21.15.110:0/1581902953 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f2778051f20 con 0x7f2780157c20 2023-06-22T18:07:12.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.585+0000 7f278557d700 1 -- 172.21.15.110:0/1581902953 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f278004c1b0 con 0x7f2780157c20 2023-06-22T18:07:12.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.586+0000 7f2766ffd700 1 -- 172.21.15.110:0/1581902953 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+13818 (secure 0 0 0) 0x7f2778018090 con 0x7f2780157c20 2023-06-22T18:07:12.590 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:07:12.590 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":16,"fsid":"cbed9da7-33df-4858-8860-5c7cacb062c0","created":"2023-06-22T18:06:59.330561+0000","modified":"2023-06-22T18:07:12.202738+0000","last_up_change":"2023-06-22T18:07:09.187638+0000","last_in_change":"2023-06-22T18:07:01.731974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-22T18:07:10.484545+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-22T18:07:10.898910+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66c18197-0807-4f09-9e82-decc44f8bb6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":117902},{"type":"v1","addr":"172.21.15.110:6801","nonce":117902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":117902},{"type":"v1","addr":"172.21.15.110:6805","nonce":117902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":117902},{"type":"v1","addr":"172.21.15.110:6813","nonce":117902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":117902},{"type":"v1","addr":"172.21.15.110:6809","nonce":117902}]},"public_addr":"172.21.15.110:6801/117902","cluster_addr":"172.21.15.110:6805/117902","heartbeat_back_addr":"172.21.15.110:6813/117902","heartbeat_front_addr":"172.21.15.110:6809/117902","state":["exists","up"]},{"osd":1,"uuid":"e2e2e368-b287-4ae9-a502-f2d0276de124","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":117903},{"type":"v1","addr":"172.21.15.110:6804","nonce":117903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":117903},{"type":"v1","addr":"172.21.15.110:6808","nonce":117903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":117903},{"type":"v1","addr":"172.21.15.110:6815","nonce":117903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":117903},{"type":"v1","addr":"172.21.15.110:6812","nonce":117903}]},"public_addr":"172.21.15.110:6804/117903","cluster_addr":"172.21.15.110:6808/117903","heartbeat_back_addr":"172.21.15.110:6815/117903","heartbeat_front_addr":"172.21.15.110:6812/117903","state":["exists","up"]},{"osd":2,"uuid":"89608384-c8bd-4032-98e6-6dd77f29f4de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":117932},{"type":"v1","addr":"172.21.15.110:6819","nonce":117932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6821","nonce":117932},{"type":"v1","addr":"172.21.15.110:6823","nonce":117932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6829","nonce":117932},{"type":"v1","addr":"172.21.15.110:6831","nonce":117932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6825","nonce":117932},{"type":"v1","addr":"172.21.15.110:6827","nonce":117932}]},"public_addr":"172.21.15.110:6819/117932","cluster_addr":"172.21.15.110:6823/117932","heartbeat_back_addr":"172.21.15.110:6831/117932","heartbeat_front_addr":"172.21.15.110:6827/117932","state":["exists","up"]},{"osd":3,"uuid":"e20af00e-ab57-4686-87b0-a60c90128569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":117933},{"type":"v1","addr":"172.21.15.110:6818","nonce":117933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":117933},{"type":"v1","addr":"172.21.15.110:6822","nonce":117933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":117933},{"type":"v1","addr":"172.21.15.110:6830","nonce":117933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":117933},{"type":"v1","addr":"172.21.15.110:6826","nonce":117933}]},"public_addr":"172.21.15.110:6818/117933","cluster_addr":"172.21.15.110:6822/117933","heartbeat_back_addr":"172.21.15.110:6830/117933","heartbeat_front_addr":"172.21.15.110:6826/117933","state":["exists","up"]},{"osd":4,"uuid":"a9142dbb-15ab-43a0-8501-5ca261a415ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":118295},{"type":"v1","addr":"172.21.15.116:6801","nonce":118295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":118295},{"type":"v1","addr":"172.21.15.116:6803","nonce":118295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":118295},{"type":"v1","addr":"172.21.15.116:6807","nonce":118295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":118295},{"type":"v1","addr":"172.21.15.116:6805","nonce":118295}]},"public_addr":"172.21.15.116:6801/118295","cluster_addr":"172.21.15.116:6803/118295","heartbeat_back_addr":"172.21.15.116:6807/118295","heartbeat_front_addr":"172.21.15.116:6805/118295","state":["exists","up"]},{"osd":5,"uuid":"e81f0aa0-d55d-4d72-937e-a1bc82b3117f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":118312},{"type":"v1","addr":"172.21.15.116:6825","nonce":118312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":118312},{"type":"v1","addr":"172.21.15.116:6827","nonce":118312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":118312},{"type":"v1","addr":"172.21.15.116:6831","nonce":118312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":118312},{"type":"v1","addr":"172.21.15.116:6829","nonce":118312}]},"public_addr":"172.21.15.116:6825/118312","cluster_addr":"172.21.15.116:6827/118312","heartbeat_back_addr":"172.21.15.116:6831/118312","heartbeat_front_addr":"172.21.15.116:6829/118312","state":["exists","up"]},{"osd":6,"uuid":"25f64c11-e6e8-46d1-b064-28743c4233df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":118296},{"type":"v1","addr":"172.21.15.116:6810","nonce":118296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":118296},{"type":"v1","addr":"172.21.15.116:6814","nonce":118296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":118296},{"type":"v1","addr":"172.21.15.116:6822","nonce":118296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":118296},{"type":"v1","addr":"172.21.15.116:6818","nonce":118296}]},"public_addr":"172.21.15.116:6810/118296","cluster_addr":"172.21.15.116:6814/118296","heartbeat_back_addr":"172.21.15.116:6822/118296","heartbeat_front_addr":"172.21.15.116:6818/118296","state":["exists","up"]},{"osd":7,"uuid":"608edcd6-65d3-4e76-bbd4-a612a6befa4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":118297},{"type":"v1","addr":"172.21.15.116:6811","nonce":118297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":118297},{"type":"v1","addr":"172.21.15.116:6815","nonce":118297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":118297},{"type":"v1","addr":"172.21.15.116:6823","nonce":118297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":118297},{"type":"v1","addr":"172.21.15.116:6819","nonce":118297}]},"public_addr":"172.21.15.116:6811/118297","cluster_addr":"172.21.15.116:6815/118297","heartbeat_back_addr":"172.21.15.116:6823/118297","heartbeat_front_addr":"172.21.15.116:6819/118297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.824390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.872398+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.931009+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.875583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.632902+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.874695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.702422+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.626714+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T18:07:12.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 -- 172.21.15.110:0/1581902953 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f276c066e00 msgr2=0x7f276c0692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f276c066e00 0x7f276c0692b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f27740099b0 tx=0x7f2774017000 comp rx=0 tx=0).stop 2023-06-22T18:07:12.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 -- 172.21.15.110:0/1581902953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2780157c20 msgr2=0x7f278015ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2780157c20 0x7f278015ae00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2778009df0 tx=0x7f277800b040 comp rx=0 tx=0).stop 2023-06-22T18:07:12.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 -- 172.21.15.110:0/1581902953 shutdown_connections 2023-06-22T18:07:12.592 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27800c8890 0x7f27801507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f276c066e00 0x7f276c0692b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2780157c20 0x7f278015ae00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 --2- 172.21.15.110:0/1581902953 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f27800cab00 0x7f2780158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 -- 172.21.15.110:0/1581902953 >> 172.21.15.110:0/1581902953 conn(0x7f278006b550 msgr2=0x7f2780151680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:12.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 -- 172.21.15.110:0/1581902953 shutdown_connections 2023-06-22T18:07:12.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.588+0000 7f278557d700 1 -- 172.21.15.110:0/1581902953 wait complete. 2023-06-22T18:07:12.604 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-06-22T18:07:10.484545+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-06-22T18:07:10.898910+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-22T18:07:12.604 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T18:07:12.724 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.721+0000 7f7ba8b4d700 1 Processor -- start 2023-06-22T18:07:12.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.721+0000 7f7ba8b4d700 1 -- start start 2023-06-22T18:07:12.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.721+0000 7f7ba8b4d700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ba406a080 0x7f7ba40ca890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.721+0000 7f7ba8b4d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7ba40cee50 0x7f7ba415a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.721+0000 7f7ba8b4d700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba4159a60 con 0x7f7ba415a710 2023-06-22T18:07:12.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.721+0000 7f7ba8b4d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba40afac0 con 0x7f7ba40cee50 2023-06-22T18:07:12.727 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.721+0000 7f7ba8b4d700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba40cadd0 con 0x7f7ba406a080 2023-06-22T18:07:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba2ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7ba40cee50 0x7f7ba415a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba2ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7ba40cee50 0x7f7ba415a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42382/0 (socket says 172.21.15.110:42382) 2023-06-22T18:07:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba2ffd700 1 -- 172.21.15.110:0/346796126 learned_addr learned my addr 172.21.15.110:0/346796126 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba37fe700 1 --2- 172.21.15.110:0/346796126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ba406a080 0x7f7ba40ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba2ffd700 1 -- 172.21.15.110:0/346796126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ba406a080 msgr2=0x7f7ba40ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba2ffd700 1 --2- 172.21.15.110:0/346796126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ba406a080 0x7f7ba40ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba2ffd700 1 -- 172.21.15.110:0/346796126 >> v1:172.21.15.116:6789/0 conn(0x7f7ba415a710 legacy=0x7f7ba415d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba2ffd700 1 -- 172.21.15.110:0/346796126 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba40cb1d0 con 0x7f7ba40cee50 2023-06-22T18:07:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba2ffd700 1 --2- 172.21.15.110:0/346796126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba40cee50 0x7f7ba415a1d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7b980099b0 tx=0x7f7b98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d34ab5ec506e61 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba27fc700 1 -- 172.21.15.110:0/346796126 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98009ea0 con 0x7f7ba40cee50 2023-06-22T18:07:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba27fc700 1 -- 172.21.15.110:0/346796126 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9800b860 con 0x7f7ba40cee50 2023-06-22T18:07:12.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.722+0000 7f7ba27fc700 1 -- 172.21.15.110:0/346796126 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9800bc00 con 0x7f7ba40cee50 2023-06-22T18:07:12.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.723+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/346796126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba40cee50 msgr2=0x7f7ba415a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.723+0000 7f7ba8b4d700 1 --2- 172.21.15.110:0/346796126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba40cee50 0x7f7ba415a1d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7b980099b0 tx=0x7f7b98009250 comp rx=0 tx=0).stop 2023-06-22T18:07:12.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.723+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/346796126 shutdown_connections 2023-06-22T18:07:12.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.723+0000 7f7ba8b4d700 1 --2- 172.21.15.110:0/346796126 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ba406a080 0x7f7ba40ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.723+0000 7f7ba8b4d700 1 --2- 172.21.15.110:0/346796126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba40cee50 0x7f7ba415a1d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.730 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.723+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/346796126 >> 172.21.15.110:0/346796126 conn(0x7f7ba40d0250 msgr2=0x7f7ba40d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:12.731 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.723+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/346796126 shutdown_connections 2023-06-22T18:07:12.731 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.723+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/346796126 wait complete. 2023-06-22T18:07:12.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.724+0000 7f7ba8b4d700 1 Processor -- start 2023-06-22T18:07:12.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.724+0000 7f7ba8b4d700 1 -- start start 2023-06-22T18:07:12.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba8b4d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba406a080 0x7f7ba40c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba8b4d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba40cee50 0x7f7ba4157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba8b4d700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ba415a710 0x7f7ba41582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba8b4d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba40c8610 con 0x7f7ba40cee50 2023-06-22T18:07:12.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba8b4d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba415b250 con 0x7f7ba406a080 2023-06-22T18:07:12.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba8b4d700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba40c7a00 con 0x7f7ba415a710 2023-06-22T18:07:12.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba2ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba40cee50 0x7f7ba4157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba406a080 0x7f7ba40c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba2ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba40cee50 0x7f7ba4157da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42394/0 (socket says 172.21.15.110:42394) 2023-06-22T18:07:12.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba2ffd700 1 -- 172.21.15.110:0/2695958216 learned_addr learned my addr 172.21.15.110:0/2695958216 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:12.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba3fff700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ba415a710 0x7f7ba41582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.734 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba2ffd700 1 -- 172.21.15.110:0/2695958216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ba415a710 msgr2=0x7f7ba41582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba2ffd700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ba415a710 0x7f7ba41582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba2ffd700 1 -- 172.21.15.110:0/2695958216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba406a080 msgr2=0x7f7ba40c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba2ffd700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba406a080 0x7f7ba40c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba2ffd700 1 -- 172.21.15.110:0/2695958216 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980094f0 con 0x7f7ba40cee50 2023-06-22T18:07:12.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba2ffd700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba40cee50 0x7f7ba4157da0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4089250 tx=0x7f7b98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:12.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7b8b7fe700 1 -- 172.21.15.110:0/2695958216 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98000d10 con 0x7f7ba40cee50 2023-06-22T18:07:12.735 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7b8b7fe700 1 -- 172.21.15.110:0/2695958216 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9800bbe0 con 0x7f7ba40cee50 2023-06-22T18:07:12.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7b8b7fe700 1 -- 172.21.15.110:0/2695958216 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98011670 con 0x7f7ba40cee50 2023-06-22T18:07:12.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/2695958216 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba40c7ce0 con 0x7f7ba40cee50 2023-06-22T18:07:12.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.725+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/2695958216 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba40c82a0 con 0x7f7ba40cee50 2023-06-22T18:07:12.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.726+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/2695958216 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba4070e80 con 0x7f7ba40cee50 2023-06-22T18:07:12.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.727+0000 7f7b8b7fe700 1 -- 172.21.15.110:0/2695958216 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f7b98011aa0 con 0x7f7ba40cee50 2023-06-22T18:07:12.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.727+0000 7f7b8b7fe700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f7b90066e00 0x7f7b900692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:12.736 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.727+0000 7f7b8b7fe700 1 -- 172.21.15.110:0/2695958216 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f7b98085900 con 0x7f7ba40cee50 2023-06-22T18:07:12.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.731+0000 7f7b8b7fe700 1 -- 172.21.15.110:0/2695958216 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f7b98052f00 con 0x7f7ba40cee50 2023-06-22T18:07:12.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.733+0000 7f7ba37fe700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f7b90066e00 0x7f7b900692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:12.737 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.733+0000 7f7ba37fe700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f7b90066e00 0x7f7b900692b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7ba40ca300 tx=0x7f7b940091a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.921+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/2695958216 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7ba40887a0 con 0x7f7ba40cee50 2023-06-22T18:07:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.922+0000 7f7b8b7fe700 1 -- 172.21.15.110:0/2695958216 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7b98056520 con 0x7f7ba40cee50 2023-06-22T18:07:12.926 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-06-22T18:07:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/2695958216 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f7b90066e00 msgr2=0x7f7b900692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f7b90066e00 0x7f7b900692b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7ba40ca300 tx=0x7f7b940091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/2695958216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba40cee50 msgr2=0x7f7ba4157da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:12.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba40cee50 0x7f7ba4157da0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4089250 tx=0x7f7b98017000 comp rx=0 tx=0).stop 2023-06-22T18:07:12.929 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/2695958216 shutdown_connections 2023-06-22T18:07:12.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba406a080 0x7f7ba40c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f7b90066e00 0x7f7b900692b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f7ba415a710 0x7f7ba41582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.930 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 --2- 172.21.15.110:0/2695958216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba40cee50 0x7f7ba4157da0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/2695958216 >> 172.21.15.110:0/2695958216 conn(0x7f7ba40d0250 msgr2=0x7f7ba415bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/2695958216 shutdown_connections 2023-06-22T18:07:12.931 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:12.924+0000 7f7ba8b4d700 1 -- 172.21.15.110:0/2695958216 wait complete. 2023-06-22T18:07:12.940 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T18:07:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.058+0000 7f8b3b556700 1 Processor -- start 2023-06-22T18:07:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.058+0000 7f8b3b556700 1 -- start start 2023-06-22T18:07:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.059+0000 7f8b3b556700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b340cf570 0x7f8b340cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.059+0000 7f8b3b556700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b340b3220 con 0x7f8b3409d7e0 2023-06-22T18:07:13.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.059+0000 7f8b3b556700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b340b37e0 con 0x7f8b3409dbb0 2023-06-22T18:07:13.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.059+0000 7f8b3b556700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3409e030 con 0x7f8b340cf570 2023-06-22T18:07:13.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.059+0000 7f8b392f2700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8b3409d7e0 0x7f8b340b3c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:42346/0 (socket says 172.21.15.110:42346) 2023-06-22T18:07:13.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.059+0000 7f8b392f2700 1 -- 172.21.15.110:0/3479341194 learned_addr learned my addr 172.21.15.110:0/3479341194 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-06-22T18:07:13.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.059+0000 7f8b39af3700 1 --2- 172.21.15.110:0/3479341194 >> v2:172.21.15.116:3300/0 conn(0x7f8b340cf570 0x7f8b340cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.059+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393348848 0 0) 0x7f8b340b3220 con 0x7f8b3409d7e0 2023-06-22T18:07:13.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b20003570 con 0x7f8b3409d7e0 2023-06-22T18:07:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 232381112 0 0) 0x7f8b20003570 con 0x7f8b3409d7e0 2023-06-22T18:07:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 >> v2:172.21.15.116:3300/0 conn(0x7f8b340cf570 msgr2=0x7f8b340cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 --2- 172.21.15.110:0/3479341194 >> v2:172.21.15.116:3300/0 conn(0x7f8b340cf570 0x7f8b340cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 >> v1:172.21.15.116:6789/0 conn(0x7f8b3409dbb0 legacy=0x7f8b340cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b340cd500 con 0x7f8b3409d7e0 2023-06-22T18:07:13.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 519278953 0 0) 0x7f8b300017f0 con 0x7f8b3409d7e0 2023-06-22T18:07:13.075 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 >> v1:172.21.15.110:6789/0 conn(0x7f8b3409d7e0 legacy=0x7f8b340b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 --2- 172.21.15.110:0/3479341194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 0x7f8b20003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 --2- 172.21.15.110:0/3479341194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b20004810 0x7f8b20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 --2- 172.21.15.110:0/3479341194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b200051c0 0x7f8b2000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3409e030 con 0x7f8b20003a60 2023-06-22T18:07:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2000c0d0 con 0x7f8b20004810 2023-06-22T18:07:13.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2000c290 con 0x7f8b200051c0 2023-06-22T18:07:13.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b392f2700 1 --2- 172.21.15.110:0/3479341194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 0x7f8b20003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b38af1700 1 --2- 172.21.15.110:0/3479341194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b20004810 0x7f8b20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b39af3700 1 --2- 172.21.15.110:0/3479341194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b200051c0 0x7f8b2000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b392f2700 1 -- 172.21.15.110:0/3479341194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b200051c0 msgr2=0x7f8b2000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:13.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b392f2700 1 --2- 172.21.15.110:0/3479341194 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b200051c0 0x7f8b2000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b392f2700 1 -- 172.21.15.110:0/3479341194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b20004810 msgr2=0x7f8b20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:13.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b392f2700 1 --2- 172.21.15.110:0/3479341194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b20004810 0x7f8b20004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b392f2700 1 -- 172.21.15.110:0/3479341194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b20003570 con 0x7f8b20003a60 2023-06-22T18:07:13.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b39af3700 1 -- 172.21.15.110:0/3479341194 reap_dead start 2023-06-22T18:07:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b392f2700 1 --2- 172.21.15.110:0/3479341194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 0x7f8b20003e70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000a930 tx=0x7f8b300021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55683e3b07fb8c42 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b300070d0 con 0x7f8b20003a60 2023-06-22T18:07:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b30007230 con 0x7f8b20003a60 2023-06-22T18:07:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.060+0000 7f8b2bfff700 1 -- 172.21.15.110:0/3479341194 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b300075d0 con 0x7f8b20003a60 2023-06-22T18:07:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.061+0000 7f8b3b556700 1 -- 172.21.15.110:0/3479341194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 msgr2=0x7f8b20003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.061+0000 7f8b3b556700 1 --2- 172.21.15.110:0/3479341194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 0x7f8b20003e70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000a930 tx=0x7f8b300021d0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.061+0000 7f8b3b556700 1 -- 172.21.15.110:0/3479341194 shutdown_connections 2023-06-22T18:07:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.061+0000 7f8b3b556700 1 --2- 172.21.15.110:0/3479341194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 0x7f8b20003e70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.061+0000 7f8b3b556700 1 -- 172.21.15.110:0/3479341194 >> 172.21.15.110:0/3479341194 conn(0x7f8b34067340 msgr2=0x7f8b34067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.061+0000 7f8b3b556700 1 -- 172.21.15.110:0/3479341194 shutdown_connections 2023-06-22T18:07:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.061+0000 7f8b3b556700 1 -- 172.21.15.110:0/3479341194 wait complete. 2023-06-22T18:07:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.062+0000 7f8b3b556700 1 Processor -- start 2023-06-22T18:07:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.062+0000 7f8b3b556700 1 -- start start 2023-06-22T18:07:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b3b556700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 0x7f8b3414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b3b556700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b340cc3d0 0x7f8b340cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b3b556700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b340c7a20 0x7f8b3414fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b3b556700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b34159fb0 con 0x7f8b20003a60 2023-06-22T18:07:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b3b556700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340cb5d0 con 0x7f8b340c7a20 2023-06-22T18:07:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b3b556700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b340cb8b0 con 0x7f8b340cc3d0 2023-06-22T18:07:13.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b392f2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 0x7f8b3414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b392f2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 0x7f8b3414e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42426/0 (socket says 172.21.15.110:42426) 2023-06-22T18:07:13.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b392f2700 1 -- 172.21.15.110:0/1352022119 learned_addr learned my addr 172.21.15.110:0/1352022119 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:13.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b39af3700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b340c7a20 0x7f8b3414fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.084 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b38af1700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b340cc3d0 0x7f8b340cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b392f2700 1 -- 172.21.15.110:0/1352022119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b340cc3d0 msgr2=0x7f8b340cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b392f2700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b340cc3d0 0x7f8b340cacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b392f2700 1 -- 172.21.15.110:0/1352022119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b340c7a20 msgr2=0x7f8b3414fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b392f2700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b340c7a20 0x7f8b3414fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.063+0000 7f8b392f2700 1 -- 172.21.15.110:0/1352022119 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b340cbb90 con 0x7f8b20003a60 2023-06-22T18:07:13.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.064+0000 7f8b392f2700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 0x7f8b3414e1f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000c0a0 tx=0x7f8b30016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:13.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.064+0000 7f8b297fa700 1 -- 172.21.15.110:0/1352022119 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b300075b0 con 0x7f8b20003a60 2023-06-22T18:07:13.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.064+0000 7f8b297fa700 1 -- 172.21.15.110:0/1352022119 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b30001250 con 0x7f8b20003a60 2023-06-22T18:07:13.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.064+0000 7f8b3b556700 1 -- 172.21.15.110:0/1352022119 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3414e730 con 0x7f8b20003a60 2023-06-22T18:07:13.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.064+0000 7f8b297fa700 1 -- 172.21.15.110:0/1352022119 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b300276a0 con 0x7f8b20003a60 2023-06-22T18:07:13.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.064+0000 7f8b3b556700 1 -- 172.21.15.110:0/1352022119 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b340c7e60 con 0x7f8b20003a60 2023-06-22T18:07:13.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.065+0000 7f8b3b556700 1 -- 172.21.15.110:0/1352022119 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b34159d80 con 0x7f8b20003a60 2023-06-22T18:07:13.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.065+0000 7f8b297fa700 1 -- 172.21.15.110:0/1352022119 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f8b30024070 con 0x7f8b20003a60 2023-06-22T18:07:13.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.066+0000 7f8b297fa700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f8b18066e00 0x7f8b180692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.066+0000 7f8b297fa700 1 -- 172.21.15.110:0/1352022119 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f8b300868d0 con 0x7f8b20003a60 2023-06-22T18:07:13.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.066+0000 7f8b38af1700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f8b18066e00 0x7f8b180692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.066+0000 7f8b38af1700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f8b18066e00 0x7f8b180692b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8b341595e0 tx=0x7f8b24007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:13.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.070+0000 7f8b297fa700 1 -- 172.21.15.110:0/1352022119 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f8b34159d80 con 0x7f8b20003a60 2023-06-22T18:07:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.258+0000 7f8b3b556700 1 -- 172.21.15.110:0/1352022119 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8b34159d80 con 0x7f8b20003a60 2023-06-22T18:07:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.259+0000 7f8b297fa700 1 -- 172.21.15.110:0/1352022119 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8b34159d80 con 0x7f8b20003a60 2023-06-22T18:07:13.262 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-06-22T18:07:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 -- 172.21.15.110:0/1352022119 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f8b18066e00 msgr2=0x7f8b180692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f8b18066e00 0x7f8b180692b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8b341595e0 tx=0x7f8b24007d30 comp rx=0 tx=0).stop 2023-06-22T18:07:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 -- 172.21.15.110:0/1352022119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 msgr2=0x7f8b3414e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 0x7f8b3414e1f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000c0a0 tx=0x7f8b30016000 comp rx=0 tx=0).stop 2023-06-22T18:07:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 -- 172.21.15.110:0/1352022119 shutdown_connections 2023-06-22T18:07:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b340c7a20 0x7f8b3414fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f8b18066e00 0x7f8b180692b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b20003a60 0x7f8b3414e1f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 --2- 172.21.15.110:0/1352022119 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f8b340cc3d0 0x7f8b340cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 -- 172.21.15.110:0/1352022119 >> 172.21.15.110:0/1352022119 conn(0x7f8b34067340 msgr2=0x7f8b340692f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 -- 172.21.15.110:0/1352022119 shutdown_connections 2023-06-22T18:07:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.261+0000 7f8b3b556700 1 -- 172.21.15.110:0/1352022119 wait complete. 2023-06-22T18:07:13.276 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-22T18:07:13.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.389+0000 7f25d1fc0700 1 Processor -- start 2023-06-22T18:07:13.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.389+0000 7f25d1fc0700 1 -- start start 2023-06-22T18:07:13.395 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25d1fc0700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25cc153880 0x7f25cc0cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25d1fc0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25cc0cd480 0x7f25cc15a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25d1fc0700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25cc15d040 con 0x7f25cc153c50 2023-06-22T18:07:13.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25d1fc0700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25cc15aab0 con 0x7f25cc0cd480 2023-06-22T18:07:13.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25d1fc0700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc15e050 con 0x7f25cc153880 2023-06-22T18:07:13.396 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25caffd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f25cc153c50 0x7f25cc0ccd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:42354/0 (socket says 172.21.15.110:42354) 2023-06-22T18:07:13.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25caffd700 1 -- 172.21.15.110:0/2907415347 learned_addr learned my addr 172.21.15.110:0/2907415347 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-06-22T18:07:13.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25cb7fe700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25cc153880 0x7f25cc0cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25cbfff700 1 --2- 172.21.15.110:0/2907415347 >> v2:172.21.15.116:3300/0 conn(0x7f25cc0cd480 0x7f25cc15a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210316207 0 0) 0x7f25cc15d040 con 0x7f25cc153c50 2023-06-22T18:07:13.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b4003570 con 0x7f25cc153c50 2023-06-22T18:07:13.397 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4024744026 0 0) 0x7f25b4003570 con 0x7f25cc153c50 2023-06-22T18:07:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25cc153880 msgr2=0x7f25cc0cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25ca7fc700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25cc153880 0x7f25cc0cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.390+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 >> v2:172.21.15.116:3300/0 conn(0x7f25cc0cd480 msgr2=0x7f25cc15a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 --2- 172.21.15.110:0/2907415347 >> v2:172.21.15.116:3300/0 conn(0x7f25cc0cd480 0x7f25cc15a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc15dbc0 con 0x7f25cc153c50 2023-06-22T18:07:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 519278953 0 0) 0x7f25c0001470 con 0x7f25cc153c50 2023-06-22T18:07:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 >> v1:172.21.15.110:6789/0 conn(0x7f25cc153c50 legacy=0x7f25cc0ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4003a60 0x7f25b4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25b4004810 0x7f25b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b40051c0 0x7f25b400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc15aab0 con 0x7f25b4003a60 2023-06-22T18:07:13.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b4004680 con 0x7f25b40051c0 2023-06-22T18:07:13.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b400c020 con 0x7f25b4004810 2023-06-22T18:07:13.399 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25cb7fe700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25cc153880 0x7f25cc0cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:13.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25caffd700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25b4004810 0x7f25b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25cbfff700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b40051c0 0x7f25b400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25cb7fe700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4003a60 0x7f25b4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25caffd700 1 -- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b40051c0 msgr2=0x7f25b400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:13.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25caffd700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b40051c0 0x7f25b400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.400 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25caffd700 1 -- 172.21.15.110:0/2907415347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4003a60 msgr2=0x7f25b4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:13.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25caffd700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4003a60 0x7f25b4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25caffd700 1 -- 172.21.15.110:0/2907415347 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b4003570 con 0x7f25b4004810 2023-06-22T18:07:13.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25cb7fe700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4003a60 0x7f25b4003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:13.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25cbfff700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b40051c0 0x7f25b400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:13.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25caffd700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25b4004810 0x7f25b4004c80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f25c000c390 tx=0x7f25cc15db30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4a36ef5390827e3 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:13.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c000cc60 con 0x7f25b4004810 2023-06-22T18:07:13.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c000cdc0 con 0x7f25b4004810 2023-06-22T18:07:13.401 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.391+0000 7f25ca7fc700 1 -- 172.21.15.110:0/2907415347 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0005b70 con 0x7f25b4004810 2023-06-22T18:07:13.402 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.392+0000 7f25d1fc0700 1 -- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25b4004810 msgr2=0x7f25b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.402 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.392+0000 7f25d1fc0700 1 --2- 172.21.15.110:0/2907415347 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25b4004810 0x7f25b4004c80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f25c000c390 tx=0x7f25cc15db30 comp rx=0 tx=0).stop 2023-06-22T18:07:13.402 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.392+0000 7f25cbfff700 1 -- 172.21.15.110:0/2907415347 reap_dead start 2023-06-22T18:07:13.402 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.392+0000 7f25d1fc0700 1 -- 172.21.15.110:0/2907415347 shutdown_connections 2023-06-22T18:07:13.402 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.392+0000 7f25d1fc0700 1 -- 172.21.15.110:0/2907415347 >> 172.21.15.110:0/2907415347 conn(0x7f25cc067340 msgr2=0x7f25cc067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:13.402 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.392+0000 7f25d1fc0700 1 -- 172.21.15.110:0/2907415347 shutdown_connections 2023-06-22T18:07:13.402 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.392+0000 7f25d1fc0700 1 -- 172.21.15.110:0/2907415347 wait complete. 2023-06-22T18:07:13.405 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.393+0000 7f25d1fc0700 1 Processor -- start 2023-06-22T18:07:13.405 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.393+0000 7f25d1fc0700 1 -- start start 2023-06-22T18:07:13.405 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25d1fc0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4004810 0x7f25cc150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.405 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25d1fc0700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25cc0c7460 0x7f25cc157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.405 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25d1fc0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25cc0c78d0 0x7f25cc157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.406 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25d1fc0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc14b1b0 con 0x7f25b4004810 2023-06-22T18:07:13.406 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25d1fc0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc1580b0 con 0x7f25cc0c78d0 2023-06-22T18:07:13.406 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25d1fc0700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc14f0a0 con 0x7f25cc0c7460 2023-06-22T18:07:13.406 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25cb7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4004810 0x7f25cc150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.406 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25cb7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4004810 0x7f25cc150140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42450/0 (socket says 172.21.15.110:42450) 2023-06-22T18:07:13.406 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25cb7fe700 1 -- 172.21.15.110:0/1627585093 learned_addr learned my addr 172.21.15.110:0/1627585093 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:13.406 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25cb7fe700 1 -- 172.21.15.110:0/1627585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25cc0c7460 msgr2=0x7f25cc157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.407 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25cb7fe700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25cc0c7460 0x7f25cc157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.407 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25cb7fe700 1 -- 172.21.15.110:0/1627585093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25cc0c78d0 msgr2=0x7f25cc157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.407 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25cbfff700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25cc0c78d0 0x7f25cc157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.407 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25cb7fe700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25cc0c78d0 0x7f25cc157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.407 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25cb7fe700 1 -- 172.21.15.110:0/1627585093 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc14f380 con 0x7f25b4004810 2023-06-22T18:07:13.407 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25cbfff700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25cc0c78d0 0x7f25cc157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-22T18:07:13.407 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.394+0000 7f25cb7fe700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4004810 0x7f25cc150140 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f25bc011dd0 tx=0x7f25bc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:13.407 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.395+0000 7f25b37fe700 1 -- 172.21.15.110:0/1627585093 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc00fcd0 con 0x7f25b4004810 2023-06-22T18:07:13.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.395+0000 7f25b37fe700 1 -- 172.21.15.110:0/1627585093 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc00fe30 con 0x7f25b4004810 2023-06-22T18:07:13.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.395+0000 7f25b37fe700 1 -- 172.21.15.110:0/1627585093 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc01d6d0 con 0x7f25b4004810 2023-06-22T18:07:13.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.395+0000 7f25d1fc0700 1 -- 172.21.15.110:0/1627585093 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25cc14f660 con 0x7f25b4004810 2023-06-22T18:07:13.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.395+0000 7f25d1fc0700 1 -- 172.21.15.110:0/1627585093 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25cc14fc20 con 0x7f25b4004810 2023-06-22T18:07:13.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.396+0000 7f25d1fc0700 1 -- 172.21.15.110:0/1627585093 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25cc04c1b0 con 0x7f25b4004810 2023-06-22T18:07:13.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.396+0000 7f25b37fe700 1 -- 172.21.15.110:0/1627585093 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f25bc01dac0 con 0x7f25b4004810 2023-06-22T18:07:13.408 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.396+0000 7f25b37fe700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f25b8066e00 0x7f25b80692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.396+0000 7f25b37fe700 1 -- 172.21.15.110:0/1627585093 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f25bc018070 con 0x7f25b4004810 2023-06-22T18:07:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.396+0000 7f25caffd700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f25b8066e00 0x7f25b80692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.397+0000 7f25caffd700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f25b8066e00 0x7f25b80692b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f25c0000c00 tx=0x7f25c0015000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.401+0000 7f25b37fe700 1 -- 172.21.15.110:0/1627585093 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f25bc052690 con 0x7f25b4004810 2023-06-22T18:07:13.631 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.628+0000 7f25d1fc0700 1 -- 172.21.15.110:0/1627585093 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25cc14fe10 con 0x7f25b4004810 2023-06-22T18:07:13.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.628+0000 7f25b37fe700 1 -- 172.21.15.110:0/1627585093 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3536 (secure 0 0 0) 0x7f25bc055cb0 con 0x7f25b4004810 2023-06-22T18:07:13.632 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:07:13.632 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1172207039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1172207039},{"type":"v1","addr":"172.21.15.116:6834","nonce":1172207039}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1504625950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1504625950},{"type":"v1","addr":"172.21.15.110:6837","nonce":1504625950}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2196824956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":2196824956},{"type":"v1","addr":"172.21.15.110:6835","nonce":2196824956}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4300,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3960914775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":3960914775},{"type":"v1","addr":"172.21.15.116:6835","nonce":3960914775}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-06-22T18:07:13.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 -- 172.21.15.110:0/1627585093 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f25b8066e00 msgr2=0x7f25b80692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f25b8066e00 0x7f25b80692b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f25c0000c00 tx=0x7f25c0015000 comp rx=0 tx=0).stop 2023-06-22T18:07:13.634 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 -- 172.21.15.110:0/1627585093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4004810 msgr2=0x7f25cc150140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4004810 0x7f25cc150140 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f25bc011dd0 tx=0x7f25bc0091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 -- 172.21.15.110:0/1627585093 shutdown_connections 2023-06-22T18:07:13.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25cc0c78d0 0x7f25cc157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f25b8066e00 0x7f25b80692b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25b4004810 0x7f25cc150140 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 --2- 172.21.15.110:0/1627585093 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f25cc0c7460 0x7f25cc157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 -- 172.21.15.110:0/1627585093 >> 172.21.15.110:0/1627585093 conn(0x7f25cc067340 msgr2=0x7f25cc0690d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:13.635 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 -- 172.21.15.110:0/1627585093 shutdown_connections 2023-06-22T18:07:13.636 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.630+0000 7f25d1fc0700 1 -- 172.21.15.110:0/1627585093 wait complete. 2023-06-22T18:07:13.636 INFO:teuthology.orchestra.run.smithi110.stderr:dumped fsmap epoch 3 2023-06-22T18:07:13.646 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T18:07:13.760 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.757+0000 7fcddf454700 1 Processor -- start 2023-06-22T18:07:13.761 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.757+0000 7fcddf454700 1 -- start start 2023-06-22T18:07:13.764 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddf454700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcdd8153880 0x7fcdd80cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddf454700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd80cd480 0x7fcdd815a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddf454700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd815d040 con 0x7fcdd8153c50 2023-06-22T18:07:13.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddf454700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd815aab0 con 0x7fcdd8153880 2023-06-22T18:07:13.765 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddf454700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd815e050 con 0x7fcdd80cd480 2023-06-22T18:07:13.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddd9f1700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd80cd480 0x7fcdd815a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddd1f0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcdd8153880 0x7fcdd80cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddc9ef700 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcdd8153c50 0x7fcdd80ccd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.110:51156/0 (socket says 172.21.15.110:51156) 2023-06-22T18:07:13.766 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddc9ef700 1 -- 172.21.15.110:0/92646085 learned_addr learned my addr 172.21.15.110:0/92646085 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-06-22T18:07:13.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddd9f1700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd80cd480 0x7fcdd815a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.110:41420/0 (socket says 172.21.15.110:41420) 2023-06-22T18:07:13.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddd9f1700 1 -- 172.21.15.110:0/92646085 >> v2:172.21.15.116:3300/0 conn(0x7fcdd8153880 msgr2=0x7fcdd80cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:13.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddd9f1700 1 --2- 172.21.15.110:0/92646085 >> v2:172.21.15.116:3300/0 conn(0x7fcdd8153880 0x7fcdd80cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.767 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddd9f1700 1 -- 172.21.15.110:0/92646085 >> v1:172.21.15.116:6789/0 conn(0x7fcdd8153c50 legacy=0x7fcdd80ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.758+0000 7fcddd9f1700 1 -- 172.21.15.110:0/92646085 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd815dbc0 con 0x7fcdd80cd480 2023-06-22T18:07:13.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.759+0000 7fcddd9f1700 1 --2- 172.21.15.110:0/92646085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd80cd480 0x7fcdd815a570 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009e60 tx=0x7fcdd0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6cd0971e31d2bb7 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:13.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.759+0000 7fcdcffff700 1 -- 172.21.15.110:0/92646085 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd000e070 con 0x7fcdd80cd480 2023-06-22T18:07:13.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.759+0000 7fcdcffff700 1 -- 172.21.15.110:0/92646085 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd000bbf0 con 0x7fcdd80cd480 2023-06-22T18:07:13.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.759+0000 7fcdcffff700 1 -- 172.21.15.110:0/92646085 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0012430 con 0x7fcdd80cd480 2023-06-22T18:07:13.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.759+0000 7fcddf454700 1 -- 172.21.15.110:0/92646085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd80cd480 msgr2=0x7fcdd815a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.759+0000 7fcddf454700 1 --2- 172.21.15.110:0/92646085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd80cd480 0x7fcdd815a570 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009e60 tx=0x7fcdd0009780 comp rx=0 tx=0).stop 2023-06-22T18:07:13.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.759+0000 7fcddf454700 1 -- 172.21.15.110:0/92646085 shutdown_connections 2023-06-22T18:07:13.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.759+0000 7fcddf454700 1 --2- 172.21.15.110:0/92646085 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd80cd480 0x7fcdd815a570 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.759+0000 7fcddf454700 1 --2- 172.21.15.110:0/92646085 >> v2:172.21.15.116:3300/0 conn(0x7fcdd8153880 0x7fcdd80cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.759+0000 7fcddf454700 1 -- 172.21.15.110:0/92646085 >> 172.21.15.110:0/92646085 conn(0x7fcdd8067340 msgr2=0x7fcdd8067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:13.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.760+0000 7fcddf454700 1 -- 172.21.15.110:0/92646085 shutdown_connections 2023-06-22T18:07:13.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.760+0000 7fcddf454700 1 -- 172.21.15.110:0/92646085 wait complete. 2023-06-22T18:07:13.769 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.761+0000 7fcddf454700 1 Processor -- start 2023-06-22T18:07:13.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.761+0000 7fcddf454700 1 -- start start 2023-06-22T18:07:13.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddf454700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdd80cd480 0x7fcdd814a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddf454700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd8153880 0x7fcdd80c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddf454700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8153c50 0x7fcdd8152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddf454700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80c9ff0 con 0x7fcdd80cd480 2023-06-22T18:07:13.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddf454700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd814ed70 con 0x7fcdd8153c50 2023-06-22T18:07:13.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddf454700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd814f050 con 0x7fcdd8153880 2023-06-22T18:07:13.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddd1f0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdd80cd480 0x7fcdd814a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddc9ef700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd8153880 0x7fcdd80c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddc9ef700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd8153880 0x7fcdd80c9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.110:41434/0 (socket says 172.21.15.110:41434) 2023-06-22T18:07:13.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddc9ef700 1 -- 172.21.15.110:0/2375300605 learned_addr learned my addr 172.21.15.110:0/2375300605 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:13.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddd9f1700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8153c50 0x7fcdd8152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddc9ef700 1 -- 172.21.15.110:0/2375300605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8153c50 msgr2=0x7fcdd8152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddc9ef700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8153c50 0x7fcdd8152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddc9ef700 1 -- 172.21.15.110:0/2375300605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdd80cd480 msgr2=0x7fcdd814a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddc9ef700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdd80cd480 0x7fcdd814a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddc9ef700 1 -- 172.21.15.110:0/2375300605 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd0009a20 con 0x7fcdd8153880 2023-06-22T18:07:13.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddd9f1700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8153c50 0x7fcdd8152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:13.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.762+0000 7fcddc9ef700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd8153880 0x7fcdd80c9630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800c590 tx=0x7fcdc8001020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:13.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.763+0000 7fcdcd7fa700 1 -- 172.21.15.110:0/2375300605 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8005b30 con 0x7fcdd8153880 2023-06-22T18:07:13.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.763+0000 7fcdcd7fa700 1 -- 172.21.15.110:0/2375300605 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc8005cb0 con 0x7fcdd8153880 2023-06-22T18:07:13.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.763+0000 7fcddf454700 1 -- 172.21.15.110:0/2375300605 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd814f330 con 0x7fcdd8153880 2023-06-22T18:07:13.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.763+0000 7fcdcd7fa700 1 -- 172.21.15.110:0/2375300605 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8007320 con 0x7fcdd8153880 2023-06-22T18:07:13.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.763+0000 7fcddf454700 1 -- 172.21.15.110:0/2375300605 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd814a5e0 con 0x7fcdd8153880 2023-06-22T18:07:13.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.764+0000 7fcddf454700 1 -- 172.21.15.110:0/2375300605 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd804c1b0 con 0x7fcdd8153880 2023-06-22T18:07:13.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.765+0000 7fcdcd7fa700 1 -- 172.21.15.110:0/2375300605 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7fcdc801b3b0 con 0x7fcdd8153880 2023-06-22T18:07:13.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.765+0000 7fcdcd7fa700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fcdbc066e00 0x7fcdbc0692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:13.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.765+0000 7fcdcd7fa700 1 -- 172.21.15.110:0/2375300605 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fcdc8083490 con 0x7fcdd8153880 2023-06-22T18:07:13.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.766+0000 7fcddd1f0700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fcdbc066e00 0x7fcdbc0692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:13.775 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.766+0000 7fcddd1f0700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fcdbc066e00 0x7fcdbc0692b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80ce8d0 tx=0x7fcdd40091a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:13.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.769+0000 7fcdcd7fa700 1 -- 172.21.15.110:0/2375300605 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fcdc8050e60 con 0x7fcdd8153880 2023-06-22T18:07:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.774+0000 7fcdcd7fa700 1 -- 172.21.15.110:0/2375300605 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fcdc805c940 con 0x7fcdd8153880 2023-06-22T18:07:13.961 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.958+0000 7fcddf454700 1 -- 172.21.15.110:0/2375300605 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdd814f850 con 0x7fcdd8153880 2023-06-22T18:07:13.962 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.959+0000 7fcdcd7fa700 1 -- 172.21.15.110:0/2375300605 <== mon.2 v2:172.21.15.116:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+13792 (secure 0 0 0) 0x7fcdd814f850 con 0x7fcdd8153880 2023-06-22T18:07:13.962 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:07:13.962 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":17,"fsid":"cbed9da7-33df-4858-8860-5c7cacb062c0","created":"2023-06-22T18:06:59.330561+0000","modified":"2023-06-22T18:07:13.213438+0000","last_up_change":"2023-06-22T18:07:09.187638+0000","last_in_change":"2023-06-22T18:07:01.731974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-22T18:07:10.484545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-22T18:07:10.898910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66c18197-0807-4f09-9e82-decc44f8bb6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":117902},{"type":"v1","addr":"172.21.15.110:6801","nonce":117902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":117902},{"type":"v1","addr":"172.21.15.110:6805","nonce":117902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":117902},{"type":"v1","addr":"172.21.15.110:6813","nonce":117902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":117902},{"type":"v1","addr":"172.21.15.110:6809","nonce":117902}]},"public_addr":"172.21.15.110:6801/117902","cluster_addr":"172.21.15.110:6805/117902","heartbeat_back_addr":"172.21.15.110:6813/117902","heartbeat_front_addr":"172.21.15.110:6809/117902","state":["exists","up"]},{"osd":1,"uuid":"e2e2e368-b287-4ae9-a502-f2d0276de124","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":117903},{"type":"v1","addr":"172.21.15.110:6804","nonce":117903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":117903},{"type":"v1","addr":"172.21.15.110:6808","nonce":117903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":117903},{"type":"v1","addr":"172.21.15.110:6815","nonce":117903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":117903},{"type":"v1","addr":"172.21.15.110:6812","nonce":117903}]},"public_addr":"172.21.15.110:6804/117903","cluster_addr":"172.21.15.110:6808/117903","heartbeat_back_addr":"172.21.15.110:6815/117903","heartbeat_front_addr":"172.21.15.110:6812/117903","state":["exists","up"]},{"osd":2,"uuid":"89608384-c8bd-4032-98e6-6dd77f29f4de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":117932},{"type":"v1","addr":"172.21.15.110:6819","nonce":117932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6821","nonce":117932},{"type":"v1","addr":"172.21.15.110:6823","nonce":117932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6829","nonce":117932},{"type":"v1","addr":"172.21.15.110:6831","nonce":117932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6825","nonce":117932},{"type":"v1","addr":"172.21.15.110:6827","nonce":117932}]},"public_addr":"172.21.15.110:6819/117932","cluster_addr":"172.21.15.110:6823/117932","heartbeat_back_addr":"172.21.15.110:6831/117932","heartbeat_front_addr":"172.21.15.110:6827/117932","state":["exists","up"]},{"osd":3,"uuid":"e20af00e-ab57-4686-87b0-a60c90128569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":117933},{"type":"v1","addr":"172.21.15.110:6818","nonce":117933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":117933},{"type":"v1","addr":"172.21.15.110:6822","nonce":117933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":117933},{"type":"v1","addr":"172.21.15.110:6830","nonce":117933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":117933},{"type":"v1","addr":"172.21.15.110:6826","nonce":117933}]},"public_addr":"172.21.15.110:6818/117933","cluster_addr":"172.21.15.110:6822/117933","heartbeat_back_addr":"172.21.15.110:6830/117933","heartbeat_front_addr":"172.21.15.110:6826/117933","state":["exists","up"]},{"osd":4,"uuid":"a9142dbb-15ab-43a0-8501-5ca261a415ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":118295},{"type":"v1","addr":"172.21.15.116:6801","nonce":118295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":118295},{"type":"v1","addr":"172.21.15.116:6803","nonce":118295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":118295},{"type":"v1","addr":"172.21.15.116:6807","nonce":118295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":118295},{"type":"v1","addr":"172.21.15.116:6805","nonce":118295}]},"public_addr":"172.21.15.116:6801/118295","cluster_addr":"172.21.15.116:6803/118295","heartbeat_back_addr":"172.21.15.116:6807/118295","heartbeat_front_addr":"172.21.15.116:6805/118295","state":["exists","up"]},{"osd":5,"uuid":"e81f0aa0-d55d-4d72-937e-a1bc82b3117f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":118312},{"type":"v1","addr":"172.21.15.116:6825","nonce":118312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":118312},{"type":"v1","addr":"172.21.15.116:6827","nonce":118312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":118312},{"type":"v1","addr":"172.21.15.116:6831","nonce":118312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":118312},{"type":"v1","addr":"172.21.15.116:6829","nonce":118312}]},"public_addr":"172.21.15.116:6825/118312","cluster_addr":"172.21.15.116:6827/118312","heartbeat_back_addr":"172.21.15.116:6831/118312","heartbeat_front_addr":"172.21.15.116:6829/118312","state":["exists","up"]},{"osd":6,"uuid":"25f64c11-e6e8-46d1-b064-28743c4233df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":118296},{"type":"v1","addr":"172.21.15.116:6810","nonce":118296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":118296},{"type":"v1","addr":"172.21.15.116:6814","nonce":118296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":118296},{"type":"v1","addr":"172.21.15.116:6822","nonce":118296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":118296},{"type":"v1","addr":"172.21.15.116:6818","nonce":118296}]},"public_addr":"172.21.15.116:6810/118296","cluster_addr":"172.21.15.116:6814/118296","heartbeat_back_addr":"172.21.15.116:6822/118296","heartbeat_front_addr":"172.21.15.116:6818/118296","state":["exists","up"]},{"osd":7,"uuid":"608edcd6-65d3-4e76-bbd4-a612a6befa4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":118297},{"type":"v1","addr":"172.21.15.116:6811","nonce":118297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":118297},{"type":"v1","addr":"172.21.15.116:6815","nonce":118297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":118297},{"type":"v1","addr":"172.21.15.116:6823","nonce":118297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":118297},{"type":"v1","addr":"172.21.15.116:6819","nonce":118297}]},"public_addr":"172.21.15.116:6811/118297","cluster_addr":"172.21.15.116:6815/118297","heartbeat_back_addr":"172.21.15.116:6823/118297","heartbeat_front_addr":"172.21.15.116:6819/118297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.824390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.872398+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.931009+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.875583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.632902+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.874695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.702422+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.626714+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T18:07:13.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.960+0000 7fcddf454700 1 -- 172.21.15.110:0/2375300605 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fcdbc066e00 msgr2=0x7fcdbc0692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.960+0000 7fcddf454700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fcdbc066e00 0x7fcdbc0692b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80ce8d0 tx=0x7fcdd40091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.960+0000 7fcddf454700 1 -- 172.21.15.110:0/2375300605 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd8153880 msgr2=0x7fcdd80c9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:13.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.960+0000 7fcddf454700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd8153880 0x7fcdd80c9630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcdc800c590 tx=0x7fcdc8001020 comp rx=0 tx=0).stop 2023-06-22T18:07:13.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.961+0000 7fcddf454700 1 -- 172.21.15.110:0/2375300605 shutdown_connections 2023-06-22T18:07:13.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.961+0000 7fcddf454700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdd8153c50 0x7fcdd8152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.961+0000 7fcddf454700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fcdbc066e00 0x7fcdbc0692b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.961+0000 7fcddf454700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdd80cd480 0x7fcdd814a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.961+0000 7fcddf454700 1 --2- 172.21.15.110:0/2375300605 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fcdd8153880 0x7fcdd80c9630 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:13.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.961+0000 7fcddf454700 1 -- 172.21.15.110:0/2375300605 >> 172.21.15.110:0/2375300605 conn(0x7fcdd8067340 msgr2=0x7fcdd80cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:13.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.961+0000 7fcddf454700 1 -- 172.21.15.110:0/2375300605 shutdown_connections 2023-06-22T18:07:13.965 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:13.961+0000 7fcddf454700 1 -- 172.21.15.110:0/2375300605 wait complete. 2023-06-22T18:07:13.976 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-06-22T18:07:10.484545+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-06-22T18:07:10.898910+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-22T18:07:13.976 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-22T18:07:14.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.090+0000 7f110314f700 1 Processor -- start 2023-06-22T18:07:14.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.090+0000 7f110314f700 1 -- start start 2023-06-22T18:07:14.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f110314f700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10fc09d7e0 0x7f10fc09dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f110314f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f10fc0b3c70 0x7f10fc0cca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f110314f700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10fc0b3220 con 0x7f10fc0ccf80 2023-06-22T18:07:14.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f110314f700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10fc0b37e0 con 0x7f10fc0b3c70 2023-06-22T18:07:14.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f110314f700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10fc0cd500 con 0x7f10fc09d7e0 2023-06-22T18:07:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f10fbfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f10fc0b3c70 0x7f10fc0cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f10fbfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f10fc0b3c70 0x7f10fc0cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42468/0 (socket says 172.21.15.110:42468) 2023-06-22T18:07:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f10fbfff700 1 -- 172.21.15.110:0/3697406764 learned_addr learned my addr 172.21.15.110:0/3697406764 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f1100eeb700 1 --2- 172.21.15.110:0/3697406764 >> v2:172.21.15.116:3300/0 conn(0x7f10fc09d7e0 0x7f10fc09dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f10fbfff700 1 -- 172.21.15.110:0/3697406764 >> v2:172.21.15.116:3300/0 conn(0x7f10fc09d7e0 msgr2=0x7f10fc09dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:14.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f10fbfff700 1 --2- 172.21.15.110:0/3697406764 >> v2:172.21.15.116:3300/0 conn(0x7f10fc09d7e0 0x7f10fc09dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f10fbfff700 1 -- 172.21.15.110:0/3697406764 >> v1:172.21.15.116:6789/0 conn(0x7f10fc0ccf80 legacy=0x7f10fc0cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:14.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f10fbfff700 1 -- 172.21.15.110:0/3697406764 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc0cd7e0 con 0x7f10fc0b3c70 2023-06-22T18:07:14.097 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f10fbfff700 1 --2- 172.21.15.110:0/3697406764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10fc0b3c70 0x7f10fc0cca40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0b6460 tx=0x7f10f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb12d1b894ecc6b server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f10fb7fe700 1 -- 172.21.15.110:0/3697406764 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f000b8b0 con 0x7f10fc0b3c70 2023-06-22T18:07:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f10fb7fe700 1 -- 172.21.15.110:0/3697406764 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f000ba10 con 0x7f10fc0b3c70 2023-06-22T18:07:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.091+0000 7f10fb7fe700 1 -- 172.21.15.110:0/3697406764 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f000bdb0 con 0x7f10fc0b3c70 2023-06-22T18:07:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.092+0000 7f110314f700 1 -- 172.21.15.110:0/3697406764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10fc0b3c70 msgr2=0x7f10fc0cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.092+0000 7f110314f700 1 --2- 172.21.15.110:0/3697406764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10fc0b3c70 0x7f10fc0cca40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0b6460 tx=0x7f10f0009250 comp rx=0 tx=0).stop 2023-06-22T18:07:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.092+0000 7f110314f700 1 -- 172.21.15.110:0/3697406764 shutdown_connections 2023-06-22T18:07:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.092+0000 7f110314f700 1 --2- 172.21.15.110:0/3697406764 >> v2:172.21.15.116:3300/0 conn(0x7f10fc09d7e0 0x7f10fc09dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.092+0000 7f110314f700 1 --2- 172.21.15.110:0/3697406764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10fc0b3c70 0x7f10fc0cca40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.092+0000 7f110314f700 1 -- 172.21.15.110:0/3697406764 >> 172.21.15.110:0/3697406764 conn(0x7f10fc067340 msgr2=0x7f10fc067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.092+0000 7f110314f700 1 -- 172.21.15.110:0/3697406764 shutdown_connections 2023-06-22T18:07:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.092+0000 7f110314f700 1 -- 172.21.15.110:0/3697406764 wait complete. 2023-06-22T18:07:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.093+0000 7f110314f700 1 Processor -- start 2023-06-22T18:07:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.093+0000 7f110314f700 1 -- start start 2023-06-22T18:07:14.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f110314f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10fc09d7e0 0x7f10fc150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f110314f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10fc0b3c70 0x7f10fc155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f110314f700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f10fc0ccf80 0x7f10fc0c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f110314f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc15b990 con 0x7f10fc0b3c70 2023-06-22T18:07:14.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f110314f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc15b060 con 0x7f10fc09d7e0 2023-06-22T18:07:14.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f110314f700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc15b340 con 0x7f10fc0ccf80 2023-06-22T18:07:14.105 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f10fbfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10fc0b3c70 0x7f10fc155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f10fbfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10fc0b3c70 0x7f10fc155780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42476/0 (socket says 172.21.15.110:42476) 2023-06-22T18:07:14.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f1100eeb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10fc09d7e0 0x7f10fc150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f10fbfff700 1 -- 172.21.15.110:0/4003414932 learned_addr learned my addr 172.21.15.110:0/4003414932 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:14.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f11016ec700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f10fc0ccf80 0x7f10fc0c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f10fbfff700 1 -- 172.21.15.110:0/4003414932 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f10fc0ccf80 msgr2=0x7f10fc0c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f10fbfff700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f10fc0ccf80 0x7f10fc0c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.106 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f10fbfff700 1 -- 172.21.15.110:0/4003414932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10fc09d7e0 msgr2=0x7f10fc150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f10fbfff700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10fc09d7e0 0x7f10fc150fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f10fbfff700 1 -- 172.21.15.110:0/4003414932 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f00094f0 con 0x7f10fc0b3c70 2023-06-22T18:07:14.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.094+0000 7f1100eeb700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10fc09d7e0 0x7f10fc150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:14.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.095+0000 7f10fbfff700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10fc0b3c70 0x7f10fc155780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009290 tx=0x7f10f0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:14.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.096+0000 7f10f8ff9700 1 -- 172.21.15.110:0/4003414932 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f0011d00 con 0x7f10fc0b3c70 2023-06-22T18:07:14.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.096+0000 7f10f8ff9700 1 -- 172.21.15.110:0/4003414932 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f0020070 con 0x7f10fc0b3c70 2023-06-22T18:07:14.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.096+0000 7f110314f700 1 -- 172.21.15.110:0/4003414932 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc15b620 con 0x7f10fc0b3c70 2023-06-22T18:07:14.107 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.096+0000 7f10f8ff9700 1 -- 172.21.15.110:0/4003414932 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f001b6a0 con 0x7f10fc0b3c70 2023-06-22T18:07:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.096+0000 7f110314f700 1 -- 172.21.15.110:0/4003414932 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc155cc0 con 0x7f10fc0b3c70 2023-06-22T18:07:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.097+0000 7f110314f700 1 -- 172.21.15.110:0/4003414932 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc15bc70 con 0x7f10fc0b3c70 2023-06-22T18:07:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.097+0000 7f10f8ff9700 1 -- 172.21.15.110:0/4003414932 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f10f0022030 con 0x7f10fc0b3c70 2023-06-22T18:07:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.098+0000 7f10f8ff9700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f10e8066e90 0x7f10e8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.098+0000 7f1100eeb700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f10e8066e90 0x7f10e8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.098+0000 7f10f8ff9700 1 -- 172.21.15.110:0/4003414932 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f10f0086080 con 0x7f10fc0b3c70 2023-06-22T18:07:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.098+0000 7f1100eeb700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f10e8066e90 0x7f10e8069340 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f10ec007990 tx=0x7f10ec0097e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.102+0000 7f10f8ff9700 1 -- 172.21.15.110:0/4003414932 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f10f00535a0 con 0x7f10fc0b3c70 2023-06-22T18:07:14.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.284+0000 7f110314f700 1 -- 172.21.15.110:0/4003414932 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f10fc155eb0 con 0x7f10fc0b3c70 2023-06-22T18:07:14.289 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.284+0000 7f10f8ff9700 1 -- 172.21.15.110:0/4003414932 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) v1 ==== 92+0+10 (secure 0 0 0) 0x7f10f0056bc0 con 0x7f10fc0b3c70 2023-06-22T18:07:14.289 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2023-06-22T18:07:14.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 -- 172.21.15.110:0/4003414932 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f10e8066e90 msgr2=0x7f10e8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f10e8066e90 0x7f10e8069340 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f10ec007990 tx=0x7f10ec0097e0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 -- 172.21.15.110:0/4003414932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10fc0b3c70 msgr2=0x7f10fc155780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10fc0b3c70 0x7f10fc155780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009290 tx=0x7f10f0016000 comp rx=0 tx=0).stop 2023-06-22T18:07:14.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 -- 172.21.15.110:0/4003414932 shutdown_connections 2023-06-22T18:07:14.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10fc09d7e0 0x7f10fc150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f10e8066e90 0x7f10e8069340 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f10fc0ccf80 0x7f10fc0c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 --2- 172.21.15.110:0/4003414932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10fc0b3c70 0x7f10fc155780 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 -- 172.21.15.110:0/4003414932 >> 172.21.15.110:0/4003414932 conn(0x7f10fc067340 msgr2=0x7f10fc069a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:14.292 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 -- 172.21.15.110:0/4003414932 shutdown_connections 2023-06-22T18:07:14.292 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.286+0000 7f110314f700 1 -- 172.21.15.110:0/4003414932 wait complete. 2023-06-22T18:07:14.301 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-22T18:07:14.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.415+0000 7fd0ba460700 1 Processor -- start 2023-06-22T18:07:14.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.415+0000 7fd0ba460700 1 -- start start 2023-06-22T18:07:14.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0ba460700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd0b409d7e0 0x7fd0b409dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0ba460700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0b40b3c70 0x7fd0b40cca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0ba460700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b40b3220 con 0x7fd0b40ccf80 2023-06-22T18:07:14.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0ba460700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b40b37e0 con 0x7fd0b409d7e0 2023-06-22T18:07:14.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0ba460700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b40cd500 con 0x7fd0b40b3c70 2023-06-22T18:07:14.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b3fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd0b409d7e0 0x7fd0b409dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b3fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd0b409d7e0 0x7fd0b409dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42482/0 (socket says 172.21.15.110:42482) 2023-06-22T18:07:14.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b3fff700 1 -- 172.21.15.110:0/840889771 learned_addr learned my addr 172.21.15.110:0/840889771 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:14.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b37fe700 1 --2- 172.21.15.110:0/840889771 >> v2:172.21.15.116:3300/0 conn(0x7fd0b40b3c70 0x7fd0b40cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b3fff700 1 -- 172.21.15.110:0/840889771 >> v2:172.21.15.116:3300/0 conn(0x7fd0b40b3c70 msgr2=0x7fd0b40cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:14.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b3fff700 1 --2- 172.21.15.110:0/840889771 >> v2:172.21.15.116:3300/0 conn(0x7fd0b40b3c70 0x7fd0b40cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b3fff700 1 -- 172.21.15.110:0/840889771 >> v1:172.21.15.116:6789/0 conn(0x7fd0b40ccf80 legacy=0x7fd0b40cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:14.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b3fff700 1 -- 172.21.15.110:0/840889771 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b40cd7e0 con 0x7fd0b409d7e0 2023-06-22T18:07:14.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b3fff700 1 --2- 172.21.15.110:0/840889771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0b409d7e0 0x7fd0b409dbb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd0b40cde20 tx=0x7fd0a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528f88ed50907870 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:14.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b2ffd700 1 -- 172.21.15.110:0/840889771 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a800b8b0 con 0x7fd0b409d7e0 2023-06-22T18:07:14.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b2ffd700 1 -- 172.21.15.110:0/840889771 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a800ba10 con 0x7fd0b409d7e0 2023-06-22T18:07:14.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.416+0000 7fd0b2ffd700 1 -- 172.21.15.110:0/840889771 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a800bdb0 con 0x7fd0b409d7e0 2023-06-22T18:07:14.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.417+0000 7fd0ba460700 1 -- 172.21.15.110:0/840889771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0b409d7e0 msgr2=0x7fd0b409dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.417+0000 7fd0ba460700 1 --2- 172.21.15.110:0/840889771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0b409d7e0 0x7fd0b409dbb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd0b40cde20 tx=0x7fd0a8009250 comp rx=0 tx=0).stop 2023-06-22T18:07:14.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.417+0000 7fd0ba460700 1 -- 172.21.15.110:0/840889771 shutdown_connections 2023-06-22T18:07:14.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.417+0000 7fd0ba460700 1 --2- 172.21.15.110:0/840889771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0b409d7e0 0x7fd0b409dbb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.417+0000 7fd0ba460700 1 --2- 172.21.15.110:0/840889771 >> v2:172.21.15.116:3300/0 conn(0x7fd0b40b3c70 0x7fd0b40cca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.417+0000 7fd0ba460700 1 -- 172.21.15.110:0/840889771 >> 172.21.15.110:0/840889771 conn(0x7fd0b4067340 msgr2=0x7fd0b4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:14.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.417+0000 7fd0ba460700 1 -- 172.21.15.110:0/840889771 shutdown_connections 2023-06-22T18:07:14.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.417+0000 7fd0ba460700 1 -- 172.21.15.110:0/840889771 wait complete. 2023-06-22T18:07:14.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.418+0000 7fd0ba460700 1 Processor -- start 2023-06-22T18:07:14.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.418+0000 7fd0ba460700 1 -- start start 2023-06-22T18:07:14.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0ba460700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b409d7e0 0x7fd0b416d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0ba460700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0b40b3c70 0x7fd0b415b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0ba460700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0b40ccf80 0x7fd0b40cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0ba460700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b40c7ee0 con 0x7fd0b40b3c70 2023-06-22T18:07:14.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0ba460700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b416d6e0 con 0x7fd0b409d7e0 2023-06-22T18:07:14.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0ba460700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b415bfb0 con 0x7fd0b40ccf80 2023-06-22T18:07:14.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0b37fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0b40b3c70 0x7fd0b415b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0b37fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0b40b3c70 0x7fd0b415b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42486/0 (socket says 172.21.15.110:42486) 2023-06-22T18:07:14.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0b37fe700 1 -- 172.21.15.110:0/1125482276 learned_addr learned my addr 172.21.15.110:0/1125482276 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:14.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0b89fd700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0b40ccf80 0x7fd0b40cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0b3fff700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b409d7e0 0x7fd0b416d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0b37fe700 1 -- 172.21.15.110:0/1125482276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0b40ccf80 msgr2=0x7fd0b40cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0b37fe700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0b40ccf80 0x7fd0b40cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0b37fe700 1 -- 172.21.15.110:0/1125482276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b409d7e0 msgr2=0x7fd0b416d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0b37fe700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b409d7e0 0x7fd0b416d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.419+0000 7fd0b37fe700 1 -- 172.21.15.110:0/1125482276 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a80094f0 con 0x7fd0b40b3c70 2023-06-22T18:07:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.420+0000 7fd0b37fe700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0b40b3c70 0x7fd0b415b560 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400d8c0 tx=0x7fd0a4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.420+0000 7fd09bfff700 1 -- 172.21.15.110:0/1125482276 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a40125f0 con 0x7fd0b40b3c70 2023-06-22T18:07:14.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.420+0000 7fd09bfff700 1 -- 172.21.15.110:0/1125482276 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a4012770 con 0x7fd0b40b3c70 2023-06-22T18:07:14.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.420+0000 7fd09bfff700 1 -- 172.21.15.110:0/1125482276 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4012b30 con 0x7fd0b40b3c70 2023-06-22T18:07:14.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.420+0000 7fd0ba460700 1 -- 172.21.15.110:0/1125482276 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b40c75b0 con 0x7fd0b40b3c70 2023-06-22T18:07:14.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.420+0000 7fd0ba460700 1 -- 172.21.15.110:0/1125482276 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b40c7b70 con 0x7fd0b40b3c70 2023-06-22T18:07:14.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.421+0000 7fd0ba460700 1 -- 172.21.15.110:0/1125482276 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b404c1b0 con 0x7fd0b40b3c70 2023-06-22T18:07:14.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.421+0000 7fd09bfff700 1 -- 172.21.15.110:0/1125482276 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fd0a401e460 con 0x7fd0b40b3c70 2023-06-22T18:07:14.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.421+0000 7fd09bfff700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fd0a0066e90 0x7fd0a0069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.421+0000 7fd09bfff700 1 -- 172.21.15.110:0/1125482276 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd0a40847c0 con 0x7fd0b40b3c70 2023-06-22T18:07:14.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.422+0000 7fd0b3fff700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fd0a0066e90 0x7fd0a0069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.422+0000 7fd0b3fff700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fd0a0066e90 0x7fd0a0069340 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd0b40cde20 tx=0x7fd0a8016000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:14.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.426+0000 7fd09bfff700 1 -- 172.21.15.110:0/1125482276 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fd0a4051ce0 con 0x7fd0b40b3c70 2023-06-22T18:07:14.611 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.607+0000 7fd0ba460700 1 -- 172.21.15.110:0/1125482276 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd0b40b3470 con 0x7fd0b40b3c70 2023-06-22T18:07:14.611 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.608+0000 7fd09bfff700 1 -- 172.21.15.110:0/1125482276 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd0a4055300 con 0x7fd0b40b3c70 2023-06-22T18:07:14.612 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2023-06-22T18:07:14.613 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 -- 172.21.15.110:0/1125482276 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fd0a0066e90 msgr2=0x7fd0a0069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.613 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fd0a0066e90 0x7fd0a0069340 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd0b40cde20 tx=0x7fd0a8016000 comp rx=0 tx=0).stop 2023-06-22T18:07:14.613 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 -- 172.21.15.110:0/1125482276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0b40b3c70 msgr2=0x7fd0b415b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0b40b3c70 0x7fd0b415b560 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400d8c0 tx=0x7fd0a4008040 comp rx=0 tx=0).stop 2023-06-22T18:07:14.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 -- 172.21.15.110:0/1125482276 shutdown_connections 2023-06-22T18:07:14.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0b409d7e0 0x7fd0b416d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fd0a0066e90 0x7fd0a0069340 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fd0b40ccf80 0x7fd0b40cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 --2- 172.21.15.110:0/1125482276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0b40b3c70 0x7fd0b415b560 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 -- 172.21.15.110:0/1125482276 >> 172.21.15.110:0/1125482276 conn(0x7fd0b4067340 msgr2=0x7fd0b4067710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:14.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 -- 172.21.15.110:0/1125482276 shutdown_connections 2023-06-22T18:07:14.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.610+0000 7fd0ba460700 1 -- 172.21.15.110:0/1125482276 wait complete. 2023-06-22T18:07:14.625 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-22T18:07:14.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.734+0000 7f964c9bb700 1 Processor -- start 2023-06-22T18:07:14.738 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.734+0000 7f964c9bb700 1 -- start start 2023-06-22T18:07:14.743 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.735+0000 7f964c9bb700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9648159200 0x7f964806a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.735+0000 7f964c9bb700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96480ca860 0x7f9648068980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.735+0000 7f964c9bb700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96480cb570 con 0x7f96480afbe0 2023-06-22T18:07:14.744 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.735+0000 7f964c9bb700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96480d00e0 con 0x7f96480ca860 2023-06-22T18:07:14.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.735+0000 7f964c9bb700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96480cacf0 con 0x7f9648159200 2023-06-22T18:07:14.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.735+0000 7f96477fe700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9648159200 0x7f964806a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.735+0000 7f963ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96480ca860 0x7f9648068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.745 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.735+0000 7f96477fe700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9648159200 0x7f964806a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3301/0 says I am v2:172.21.15.110:41472/0 (socket says 172.21.15.110:41472) 2023-06-22T18:07:14.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.735+0000 7f96477fe700 1 -- 172.21.15.110:0/3731301880 learned_addr learned my addr 172.21.15.110:0/3731301880 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:14.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.735+0000 7f963ffff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96480ca860 0x7f9648068980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.110:53216/0 (socket says 172.21.15.110:53216) 2023-06-22T18:07:14.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f9646ffd700 1 -- 172.21.15.110:0/3731301880 <== mon.1 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099516095 0 0) 0x7f96480cb570 con 0x7f96480afbe0 2023-06-22T18:07:14.746 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f9646ffd700 1 -- 172.21.15.110:0/3731301880 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9634003570 con 0x7f96480afbe0 2023-06-22T18:07:14.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f96477fe700 1 -- 172.21.15.110:0/3731301880 >> v2:172.21.15.116:3300/0 conn(0x7f96480ca860 msgr2=0x7f9648068980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:14.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f96477fe700 1 --2- 172.21.15.110:0/3731301880 >> v2:172.21.15.116:3300/0 conn(0x7f96480ca860 0x7f9648068980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f96477fe700 1 -- 172.21.15.110:0/3731301880 >> v1:172.21.15.116:6789/0 conn(0x7f96480afbe0 legacy=0x7f964815ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.747 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f96477fe700 1 -- 172.21.15.110:0/3731301880 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96480ce5f0 con 0x7f9648159200 2023-06-22T18:07:14.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f963ffff700 1 --2- 172.21.15.110:0/3731301880 >> v2:172.21.15.116:3300/0 conn(0x7f96480ca860 0x7f9648068980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:14.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f96477fe700 1 --2- 172.21.15.110:0/3731301880 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9648159200 0x7f964806a780 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f96380099b0 tx=0x7f9638009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc1daca44fef8753 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:14.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f9646ffd700 1 -- 172.21.15.110:0/3731301880 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963800b710 con 0x7f9648159200 2023-06-22T18:07:14.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f9646ffd700 1 -- 172.21.15.110:0/3731301880 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963800b870 con 0x7f9648159200 2023-06-22T18:07:14.748 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f9646ffd700 1 -- 172.21.15.110:0/3731301880 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963800bc10 con 0x7f9648159200 2023-06-22T18:07:14.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f964c9bb700 1 -- 172.21.15.110:0/3731301880 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9648159200 msgr2=0x7f964806a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f964c9bb700 1 --2- 172.21.15.110:0/3731301880 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9648159200 0x7f964806a780 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f96380099b0 tx=0x7f9638009250 comp rx=0 tx=0).stop 2023-06-22T18:07:14.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f964c9bb700 1 -- 172.21.15.110:0/3731301880 shutdown_connections 2023-06-22T18:07:14.749 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f964c9bb700 1 --2- 172.21.15.110:0/3731301880 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f9648159200 0x7f964806a780 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f964c9bb700 1 --2- 172.21.15.110:0/3731301880 >> v2:172.21.15.116:3300/0 conn(0x7f96480ca860 0x7f9648068980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.736+0000 7f964c9bb700 1 -- 172.21.15.110:0/3731301880 >> 172.21.15.110:0/3731301880 conn(0x7f96480d0220 msgr2=0x7f96480d0620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.737+0000 7f964c9bb700 1 -- 172.21.15.110:0/3731301880 shutdown_connections 2023-06-22T18:07:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.737+0000 7f964c9bb700 1 -- 172.21.15.110:0/3731301880 wait complete. 2023-06-22T18:07:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f964c9bb700 1 Processor -- start 2023-06-22T18:07:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f964c9bb700 1 -- start start 2023-06-22T18:07:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f964c9bb700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96480afbe0 0x7f9648150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f964c9bb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96480ca860 0x7f96480c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f964c9bb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9648159200 0x7f96480b4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f964c9bb700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96480b3160 con 0x7f96480ca860 2023-06-22T18:07:14.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f964c9bb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96480b3440 con 0x7f9648159200 2023-06-22T18:07:14.752 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f964c9bb700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96480b3720 con 0x7f96480afbe0 2023-06-22T18:07:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f963ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96480ca860 0x7f96480c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f963ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96480ca860 0x7f96480c8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42500/0 (socket says 172.21.15.110:42500) 2023-06-22T18:07:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f963ffff700 1 -- 172.21.15.110:0/1498930149 learned_addr learned my addr 172.21.15.110:0/1498930149 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f96477fe700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96480afbe0 0x7f9648150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.738+0000 7f9647fff700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9648159200 0x7f96480b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f96477fe700 1 -- 172.21.15.110:0/1498930149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9648159200 msgr2=0x7f96480b4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f96477fe700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9648159200 0x7f96480b4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f96477fe700 1 -- 172.21.15.110:0/1498930149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96480ca860 msgr2=0x7f96480c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.755 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f96477fe700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96480ca860 0x7f96480c8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.755 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f96477fe700 1 -- 172.21.15.110:0/1498930149 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9630009420 con 0x7f96480afbe0 2023-06-22T18:07:14.755 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f9647fff700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9648159200 0x7f96480b4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:14.755 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f963ffff700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96480ca860 0x7f96480c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-22T18:07:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f96477fe700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96480afbe0 0x7f9648150210 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f96480cdfc0 tx=0x7f9638017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f963f7fe700 1 -- 172.21.15.110:0/1498930149 <== mon.2 v2:172.21.15.116:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638000d10 con 0x7f96480afbe0 2023-06-22T18:07:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f963f7fe700 1 -- 172.21.15.110:0/1498930149 <== mon.2 v2:172.21.15.116:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963800bbf0 con 0x7f96480afbe0 2023-06-22T18:07:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f964c9bb700 1 -- 172.21.15.110:0/1498930149 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96380094f0 con 0x7f96480afbe0 2023-06-22T18:07:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f963f7fe700 1 -- 172.21.15.110:0/1498930149 <== mon.2 v2:172.21.15.116:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963801a6a0 con 0x7f96480afbe0 2023-06-22T18:07:14.757 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.739+0000 7f964c9bb700 1 -- 172.21.15.110:0/1498930149 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96480b3ce0 con 0x7f96480afbe0 2023-06-22T18:07:14.757 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.740+0000 7f964c9bb700 1 -- 172.21.15.110:0/1498930149 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964804c1b0 con 0x7f96480afbe0 2023-06-22T18:07:14.757 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.740+0000 7f963f7fe700 1 -- 172.21.15.110:0/1498930149 <== mon.2 v2:172.21.15.116:3301/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f963801aa90 con 0x7f96480afbe0 2023-06-22T18:07:14.758 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.741+0000 7f963f7fe700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f9628066e90 0x7f9628069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:14.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.741+0000 7f963f7fe700 1 -- 172.21.15.110:0/1498930149 <== mon.2 v2:172.21.15.116:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f9638085200 con 0x7f96480afbe0 2023-06-22T18:07:14.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.741+0000 7f963ffff700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f9628066e90 0x7f9628069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:14.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.741+0000 7f963ffff700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f9628066e90 0x7f9628069340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f96480cc470 tx=0x7f96300091a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:14.759 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.745+0000 7f963f7fe700 1 -- 172.21.15.110:0/1498930149 <== mon.2 v2:172.21.15.116:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f9638052720 con 0x7f96480afbe0 2023-06-22T18:07:14.936 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.933+0000 7f964c9bb700 1 -- 172.21.15.110:0/1498930149 --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f96480b3ed0 con 0x7f96480afbe0 2023-06-22T18:07:14.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.934+0000 7f963f7fe700 1 -- 172.21.15.110:0/1498930149 <== mon.2 v2:172.21.15.116:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+126 (secure 0 0 0) 0x7f9638055d40 con 0x7f96480afbe0 2023-06-22T18:07:14.938 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:07:14.938 INFO:teuthology.orchestra.run.smithi110.stdout:[ 2023-06-22T18:07:14.938 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-06-22T18:07:14.938 INFO:teuthology.orchestra.run.smithi110.stdout: "poolnum": 1, 2023-06-22T18:07:14.938 INFO:teuthology.orchestra.run.smithi110.stdout: "poolname": "rbd" 2023-06-22T18:07:14.938 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-06-22T18:07:14.938 INFO:teuthology.orchestra.run.smithi110.stdout: { 2023-06-22T18:07:14.939 INFO:teuthology.orchestra.run.smithi110.stdout: "poolnum": 2, 2023-06-22T18:07:14.939 INFO:teuthology.orchestra.run.smithi110.stdout: "poolname": ".mgr" 2023-06-22T18:07:14.939 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-06-22T18:07:14.939 INFO:teuthology.orchestra.run.smithi110.stdout:] 2023-06-22T18:07:14.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 -- 172.21.15.110:0/1498930149 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f9628066e90 msgr2=0x7f9628069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f9628066e90 0x7f9628069340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f96480cc470 tx=0x7f96300091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 -- 172.21.15.110:0/1498930149 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96480afbe0 msgr2=0x7f9648150210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:14.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96480afbe0 0x7f9648150210 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f96480cdfc0 tx=0x7f9638017000 comp rx=0 tx=0).stop 2023-06-22T18:07:14.940 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 -- 172.21.15.110:0/1498930149 shutdown_connections 2023-06-22T18:07:14.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9648159200 0x7f96480b4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f9628066e90 0x7f9628069340 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f96480afbe0 0x7f9648150210 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 --2- 172.21.15.110:0/1498930149 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96480ca860 0x7f96480c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:14.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 -- 172.21.15.110:0/1498930149 >> 172.21.15.110:0/1498930149 conn(0x7f96480d0220 msgr2=0x7f9648153f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:14.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 -- 172.21.15.110:0/1498930149 shutdown_connections 2023-06-22T18:07:14.941 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:14.936+0000 7f964c9bb700 1 -- 172.21.15.110:0/1498930149 wait complete. 2023-06-22T18:07:14.951 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-22T18:07:14.951 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-22T18:07:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.065+0000 7f32833a0700 1 Processor -- start 2023-06-22T18:07:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.065+0000 7f32833a0700 1 -- start start 2023-06-22T18:07:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.066+0000 7f32833a0700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f327c06db90 0x7f327c0b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.066+0000 7f32833a0700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f327c0cc1b0 0x7f327c158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.066+0000 7f32833a0700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327c0cbd20 con 0x7f327c06ea60 2023-06-22T18:07:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.066+0000 7f32833a0700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f327c0b24b0 con 0x7f327c0cc1b0 2023-06-22T18:07:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.066+0000 7f32833a0700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f327c0cb6d0 con 0x7f327c06db90 2023-06-22T18:07:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.066+0000 7f3281b9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f327c0cc1b0 0x7f327c158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.066+0000 7f3282b9f700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f327c06ea60 0x7f327c15cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:42370/0 (socket says 172.21.15.110:42370) 2023-06-22T18:07:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.066+0000 7f3282b9f700 1 -- 172.21.15.110:0/3937299328 learned_addr learned my addr 172.21.15.110:0/3937299328 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-06-22T18:07:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f328239e700 1 --2- 172.21.15.110:0/3937299328 >> v2:172.21.15.116:3300/0 conn(0x7f327c06db90 0x7f327c0b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f3281b9d700 1 -- 172.21.15.110:0/3937299328 >> v2:172.21.15.116:3300/0 conn(0x7f327c06db90 msgr2=0x7f327c0b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f3281b9d700 1 --2- 172.21.15.110:0/3937299328 >> v2:172.21.15.116:3300/0 conn(0x7f327c06db90 0x7f327c0b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f3281b9d700 1 -- 172.21.15.110:0/3937299328 >> v1:172.21.15.110:6789/0 conn(0x7f327c06ea60 legacy=0x7f327c15cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f328139c700 1 -- 172.21.15.110:0/3937299328 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581487904 0 0) 0x7f327c0cbd20 con 0x7f327c06ea60 2023-06-22T18:07:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f3281b9d700 1 -- 172.21.15.110:0/3937299328 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c0cb9b0 con 0x7f327c0cc1b0 2023-06-22T18:07:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f3281b9d700 1 --2- 172.21.15.110:0/3937299328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f327c0cc1b0 0x7f327c158ef0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f327c064050 tx=0x7f326c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19789a7d13eafae9 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f328139c700 1 -- 172.21.15.110:0/3937299328 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c00b6b0 con 0x7f327c0cc1b0 2023-06-22T18:07:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f328139c700 1 -- 172.21.15.110:0/3937299328 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c00b810 con 0x7f327c0cc1b0 2023-06-22T18:07:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f328139c700 1 -- 172.21.15.110:0/3937299328 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c00bbb0 con 0x7f327c0cc1b0 2023-06-22T18:07:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f32833a0700 1 -- 172.21.15.110:0/3937299328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f327c0cc1b0 msgr2=0x7f327c158ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.067+0000 7f32833a0700 1 --2- 172.21.15.110:0/3937299328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f327c0cc1b0 0x7f327c158ef0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f327c064050 tx=0x7f326c0091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.068+0000 7f32833a0700 1 -- 172.21.15.110:0/3937299328 shutdown_connections 2023-06-22T18:07:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.068+0000 7f32833a0700 1 --2- 172.21.15.110:0/3937299328 >> v2:172.21.15.116:3300/0 conn(0x7f327c06db90 0x7f327c0b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.068+0000 7f32833a0700 1 --2- 172.21.15.110:0/3937299328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f327c0cc1b0 0x7f327c158ef0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.068+0000 7f32833a0700 1 -- 172.21.15.110:0/3937299328 >> 172.21.15.110:0/3937299328 conn(0x7f327c0cd0f0 msgr2=0x7f327c0b15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.068+0000 7f32833a0700 1 -- 172.21.15.110:0/3937299328 shutdown_connections 2023-06-22T18:07:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.068+0000 7f32833a0700 1 -- 172.21.15.110:0/3937299328 wait complete. 2023-06-22T18:07:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.069+0000 7f32833a0700 1 Processor -- start 2023-06-22T18:07:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.069+0000 7f32833a0700 1 -- start start 2023-06-22T18:07:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.069+0000 7f32833a0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c06db90 0x7f327c158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.069+0000 7f32833a0700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f327c06ea60 0x7f327c169690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.069+0000 7f32833a0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f327c0cc1b0 0x7f327c0c80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.069+0000 7f32833a0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c159eb0 con 0x7f327c0cc1b0 2023-06-22T18:07:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.069+0000 7f32833a0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c1592a0 con 0x7f327c06db90 2023-06-22T18:07:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.069+0000 7f32833a0700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327c159580 con 0x7f327c06ea60 2023-06-22T18:07:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.069+0000 7f3282b9f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f327c0cc1b0 0x7f327c0c80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.069+0000 7f3282b9f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f327c0cc1b0 0x7f327c0c80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42516/0 (socket says 172.21.15.110:42516) 2023-06-22T18:07:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.069+0000 7f3282b9f700 1 -- 172.21.15.110:0/2110784707 learned_addr learned my addr 172.21.15.110:0/2110784707 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f328239e700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c06db90 0x7f327c158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f3281b9d700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f327c06ea60 0x7f327c169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f3282b9f700 1 -- 172.21.15.110:0/2110784707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f327c06ea60 msgr2=0x7f327c169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f3282b9f700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f327c06ea60 0x7f327c169690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f3282b9f700 1 -- 172.21.15.110:0/2110784707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c06db90 msgr2=0x7f327c158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f3282b9f700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c06db90 0x7f327c158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f3282b9f700 1 -- 172.21.15.110:0/2110784707 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c009420 con 0x7f327c0cc1b0 2023-06-22T18:07:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f328239e700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c06db90 0x7f327c158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T18:07:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f3282b9f700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f327c0cc1b0 0x7f327c0c80a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f327400cc90 tx=0x7f3274001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f32727fc700 1 -- 172.21.15.110:0/2110784707 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327400a960 con 0x7f327c0cc1b0 2023-06-22T18:07:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f32727fc700 1 -- 172.21.15.110:0/2110784707 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327400aac0 con 0x7f327c0cc1b0 2023-06-22T18:07:15.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f32727fc700 1 -- 172.21.15.110:0/2110784707 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327400ae80 con 0x7f327c0cc1b0 2023-06-22T18:07:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f32833a0700 1 -- 172.21.15.110:0/2110784707 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327c159860 con 0x7f327c0cc1b0 2023-06-22T18:07:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.070+0000 7f32833a0700 1 -- 172.21.15.110:0/2110784707 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327c0c85e0 con 0x7f327c0cc1b0 2023-06-22T18:07:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.071+0000 7f32833a0700 1 -- 172.21.15.110:0/2110784707 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f327c15a020 con 0x7f327c0cc1b0 2023-06-22T18:07:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.071+0000 7f32727fc700 1 -- 172.21.15.110:0/2110784707 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f3274007300 con 0x7f327c0cc1b0 2023-06-22T18:07:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.072+0000 7f32727fc700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f3260066e90 0x7f3260069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.072+0000 7f32727fc700 1 -- 172.21.15.110:0/2110784707 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3274084610 con 0x7f327c0cc1b0 2023-06-22T18:07:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.072+0000 7f328239e700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f3260066e90 0x7f3260069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.072+0000 7f328239e700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f3260066e90 0x7f3260069340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f327800ab70 tx=0x7f3278008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.076+0000 7f32727fc700 1 -- 172.21.15.110:0/2110784707 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f3274051bb0 con 0x7f327c0cc1b0 2023-06-22T18:07:15.264 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:15.261+0000 7f32833a0700 1 -- 172.21.15.110:0/2110784707 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f327c04c1b0 con 0x7f327c0cc1b0 2023-06-22T18:07:16.231 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.228+0000 7f32727fc700 1 -- 172.21.15.110:0/2110784707 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f32740551d0 con 0x7f327c0cc1b0 2023-06-22T18:07:16.235 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.232+0000 7f32833a0700 1 -- 172.21.15.110:0/2110784707 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f3260066e90 msgr2=0x7f3260069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:16.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.232+0000 7f32833a0700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f3260066e90 0x7f3260069340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f327800ab70 tx=0x7f3278008040 comp rx=0 tx=0).stop 2023-06-22T18:07:16.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.232+0000 7f32833a0700 1 -- 172.21.15.110:0/2110784707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f327c0cc1b0 msgr2=0x7f327c0c80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:16.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.232+0000 7f32833a0700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f327c0cc1b0 0x7f327c0c80a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f327400cc90 tx=0x7f3274001a80 comp rx=0 tx=0).stop 2023-06-22T18:07:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.233+0000 7f32833a0700 1 -- 172.21.15.110:0/2110784707 shutdown_connections 2023-06-22T18:07:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.233+0000 7f32833a0700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f327c06db90 0x7f327c158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.233+0000 7f32833a0700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f3260066e90 0x7f3260069340 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.233+0000 7f32833a0700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f327c0cc1b0 0x7f327c0c80a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.233+0000 7f32833a0700 1 --2- 172.21.15.110:0/2110784707 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f327c06ea60 0x7f327c169690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:16.237 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.233+0000 7f32833a0700 1 -- 172.21.15.110:0/2110784707 >> 172.21.15.110:0/2110784707 conn(0x7f327c0cd0f0 msgr2=0x7f327c155070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:16.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.233+0000 7f32833a0700 1 -- 172.21.15.110:0/2110784707 shutdown_connections 2023-06-22T18:07:16.238 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.233+0000 7f32833a0700 1 -- 172.21.15.110:0/2110784707 wait complete. 2023-06-22T18:07:16.239 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_metadata' created 2023-06-22T18:07:16.252 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-22T18:07:16.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.375+0000 7fbb2a7c7700 1 Processor -- start 2023-06-22T18:07:16.379 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.376+0000 7fbb2a7c7700 1 -- start start 2023-06-22T18:07:16.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.376+0000 7fbb2a7c7700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb24142a20 0x7fbb240b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:16.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.376+0000 7fbb2a7c7700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb240b2cc0 con 0x7fbb24142df0 2023-06-22T18:07:16.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.376+0000 7fbb2a7c7700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb240b0220 con 0x7fbb240cec60 2023-06-22T18:07:16.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.376+0000 7fbb2a7c7700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb240b2f60 con 0x7fbb24142a20 2023-06-22T18:07:16.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.376+0000 7fbb28fc4700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fbb24142df0 0x7fbb240af9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:42376/0 (socket says 172.21.15.110:42376) 2023-06-22T18:07:16.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.376+0000 7fbb28fc4700 1 -- 172.21.15.110:0/517180198 learned_addr learned my addr 172.21.15.110:0/517180198 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-06-22T18:07:16.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.377+0000 7fbb297c5700 1 --2- 172.21.15.110:0/517180198 >> v2:172.21.15.116:3300/0 conn(0x7fbb24142a20 0x7fbb240b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:16.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.377+0000 7fbb1bfff700 1 -- 172.21.15.110:0/517180198 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356820392 0 0) 0x7fbb240b2cc0 con 0x7fbb24142df0 2023-06-22T18:07:16.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.377+0000 7fbb1bfff700 1 -- 172.21.15.110:0/517180198 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb10003570 con 0x7fbb24142df0 2023-06-22T18:07:16.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.377+0000 7fbb297c5700 1 -- 172.21.15.110:0/517180198 >> v1:172.21.15.116:6789/0 conn(0x7fbb240cec60 legacy=0x7fbb240b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:16.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.377+0000 7fbb1bfff700 1 -- 172.21.15.110:0/517180198 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3375286017 0 0) 0x7fbb10003570 con 0x7fbb24142df0 2023-06-22T18:07:16.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.377+0000 7fbb297c5700 1 -- 172.21.15.110:0/517180198 >> v1:172.21.15.110:6789/0 conn(0x7fbb24142df0 legacy=0x7fbb240af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:16.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.377+0000 7fbb297c5700 1 -- 172.21.15.110:0/517180198 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb240b3240 con 0x7fbb24142a20 2023-06-22T18:07:16.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.377+0000 7fbb297c5700 1 --2- 172.21.15.110:0/517180198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb24142a20 0x7fbb240b3a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009860 tx=0x7fbb200091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd4c84795fec4d94 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:16.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.377+0000 7fbb1bfff700 1 -- 172.21.15.110:0/517180198 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20011630 con 0x7fbb24142a20 2023-06-22T18:07:16.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.377+0000 7fbb1bfff700 1 -- 172.21.15.110:0/517180198 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb20011790 con 0x7fbb24142a20 2023-06-22T18:07:16.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.377+0000 7fbb1bfff700 1 -- 172.21.15.110:0/517180198 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20011b30 con 0x7fbb24142a20 2023-06-22T18:07:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.378+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/517180198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb24142a20 msgr2=0x7fbb240b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.378+0000 7fbb2a7c7700 1 --2- 172.21.15.110:0/517180198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb24142a20 0x7fbb240b3a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbb20009860 tx=0x7fbb200091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.378+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/517180198 shutdown_connections 2023-06-22T18:07:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.378+0000 7fbb2a7c7700 1 --2- 172.21.15.110:0/517180198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb24142a20 0x7fbb240b3a40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.378+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/517180198 >> 172.21.15.110:0/517180198 conn(0x7fbb24070220 msgr2=0x7fbb24070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.378+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/517180198 shutdown_connections 2023-06-22T18:07:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.378+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/517180198 wait complete. 2023-06-22T18:07:16.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.379+0000 7fbb2a7c7700 1 Processor -- start 2023-06-22T18:07:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.379+0000 7fbb2a7c7700 1 -- start start 2023-06-22T18:07:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb2a7c7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb240cec60 0x7fbb2413a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb2a7c7700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb24142df0 0x7fbb24158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb2a7c7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb2415bae0 0x7fbb2413b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb2a7c7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2415c710 con 0x7fbb240cec60 2023-06-22T18:07:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb2a7c7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2413b020 con 0x7fbb2415bae0 2023-06-22T18:07:16.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb2a7c7700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2415a2f0 con 0x7fbb24142df0 2023-06-22T18:07:16.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb297c5700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb240cec60 0x7fbb2413a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:16.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb297c5700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb240cec60 0x7fbb2413a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42532/0 (socket says 172.21.15.110:42532) 2023-06-22T18:07:16.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb297c5700 1 -- 172.21.15.110:0/2737580834 learned_addr learned my addr 172.21.15.110:0/2737580834 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:16.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb29fc6700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb2415bae0 0x7fbb2413b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:16.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb28fc4700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb24142df0 0x7fbb24158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:16.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb297c5700 1 -- 172.21.15.110:0/2737580834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb24142df0 msgr2=0x7fbb24158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:16.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb297c5700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb24142df0 0x7fbb24158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:16.387 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb297c5700 1 -- 172.21.15.110:0/2737580834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb2415bae0 msgr2=0x7fbb2413b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:16.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb297c5700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb2415bae0 0x7fbb2413b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:16.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.380+0000 7fbb297c5700 1 -- 172.21.15.110:0/2737580834 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb20009420 con 0x7fbb240cec60 2023-06-22T18:07:16.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.381+0000 7fbb297c5700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb240cec60 0x7fbb2413a940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbb20002000 tx=0x7fbb20016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:16.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.381+0000 7fbb197fa700 1 -- 172.21.15.110:0/2737580834 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20011b10 con 0x7fbb240cec60 2023-06-22T18:07:16.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.381+0000 7fbb197fa700 1 -- 172.21.15.110:0/2737580834 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb20006af0 con 0x7fbb240cec60 2023-06-22T18:07:16.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.381+0000 7fbb197fa700 1 -- 172.21.15.110:0/2737580834 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2001a6a0 con 0x7fbb240cec60 2023-06-22T18:07:16.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.381+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/2737580834 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb241599c0 con 0x7fbb240cec60 2023-06-22T18:07:16.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.381+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/2737580834 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb24159f80 con 0x7fbb240cec60 2023-06-22T18:07:16.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.382+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/2737580834 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb240ad6b0 con 0x7fbb240cec60 2023-06-22T18:07:16.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.382+0000 7fbb197fa700 1 -- 172.21.15.110:0/2737580834 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fbb2001aa90 con 0x7fbb240cec60 2023-06-22T18:07:16.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.382+0000 7fbb197fa700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fbb08066e90 0x7fbb08069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:16.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.383+0000 7fbb28fc4700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fbb08066e90 0x7fbb08069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:16.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.383+0000 7fbb197fa700 1 -- 172.21.15.110:0/2737580834 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5576+0+0 (secure 0 0 0) 0x7fbb20085d50 con 0x7fbb240cec60 2023-06-22T18:07:16.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.383+0000 7fbb28fc4700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fbb08066e90 0x7fbb08069340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbb2413f570 tx=0x7fbb14007bb0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:16.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.387+0000 7fbb197fa700 1 -- 172.21.15.110:0/2737580834 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fbb20089030 con 0x7fbb240cec60 2023-06-22T18:07:16.575 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:16.572+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/2737580834 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbb240b3520 con 0x7fbb240cec60 2023-06-22T18:07:17.236 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.233+0000 7fbb197fa700 1 -- 172.21.15.110:0/2737580834 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbb200531b0 con 0x7fbb240cec60 2023-06-22T18:07:17.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.237+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/2737580834 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fbb08066e90 msgr2=0x7fbb08069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.237+0000 7fbb2a7c7700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fbb08066e90 0x7fbb08069340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbb2413f570 tx=0x7fbb14007bb0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.237+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/2737580834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb240cec60 msgr2=0x7fbb2413a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.237+0000 7fbb2a7c7700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb240cec60 0x7fbb2413a940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbb20002000 tx=0x7fbb20016000 comp rx=0 tx=0).stop 2023-06-22T18:07:17.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.238+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/2737580834 shutdown_connections 2023-06-22T18:07:17.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.238+0000 7fbb2a7c7700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb2415bae0 0x7fbb2413b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.242 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.238+0000 7fbb2a7c7700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fbb08066e90 0x7fbb08069340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.243 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.238+0000 7fbb2a7c7700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb240cec60 0x7fbb2413a940 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.243 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.238+0000 7fbb2a7c7700 1 --2- 172.21.15.110:0/2737580834 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fbb24142df0 0x7fbb24158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.243 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.238+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/2737580834 >> 172.21.15.110:0/2737580834 conn(0x7fbb24070220 msgr2=0x7fbb240b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:17.243 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.239+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/2737580834 shutdown_connections 2023-06-22T18:07:17.244 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.240+0000 7fbb2a7c7700 1 -- 172.21.15.110:0/2737580834 wait complete. 2023-06-22T18:07:17.246 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'cephfs_data' created 2023-06-22T18:07:17.263 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-22T18:07:17.380 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.377+0000 7f01f10aa700 1 Processor -- start 2023-06-22T18:07:17.381 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.377+0000 7f01f10aa700 1 -- start start 2023-06-22T18:07:17.382 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f10aa700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01ec0cee50 0x7f01ec15a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f10aa700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f01ec15a710 0x7f01ec15d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f10aa700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ec159a60 con 0x7f01ec06a080 2023-06-22T18:07:17.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f10aa700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec0afac0 con 0x7f01ec15a710 2023-06-22T18:07:17.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f10aa700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ec0cb1d0 con 0x7f01ec0cee50 2023-06-22T18:07:17.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f08a9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f01ec15a710 0x7f01ec15d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:17.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f08a9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f01ec15a710 0x7f01ec15d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42546/0 (socket says 172.21.15.110:42546) 2023-06-22T18:07:17.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f08a9700 1 -- 172.21.15.110:0/2486461218 learned_addr learned my addr 172.21.15.110:0/2486461218 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:17.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f08a9700 1 -- 172.21.15.110:0/2486461218 >> v2:172.21.15.116:3300/0 conn(0x7f01ec0cee50 msgr2=0x7f01ec15a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:17.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f08a9700 1 --2- 172.21.15.110:0/2486461218 >> v2:172.21.15.116:3300/0 conn(0x7f01ec0cee50 0x7f01ec15a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f08a9700 1 -- 172.21.15.110:0/2486461218 >> v1:172.21.15.116:6789/0 conn(0x7f01ec06a080 legacy=0x7f01ec0ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:17.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f08a9700 1 -- 172.21.15.110:0/2486461218 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec0cb4b0 con 0x7f01ec15a710 2023-06-22T18:07:17.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01f08a9700 1 --2- 172.21.15.110:0/2486461218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01ec15a710 0x7f01ec15d060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0d0030 tx=0x7f01e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22733e23ee1cc765 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:17.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.378+0000 7f01eaffd700 1 -- 172.21.15.110:0/2486461218 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e400e070 con 0x7f01ec15a710 2023-06-22T18:07:17.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.379+0000 7f01eaffd700 1 -- 172.21.15.110:0/2486461218 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e400ba10 con 0x7f01ec15a710 2023-06-22T18:07:17.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.379+0000 7f01eaffd700 1 -- 172.21.15.110:0/2486461218 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e40092e0 con 0x7f01ec15a710 2023-06-22T18:07:17.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.379+0000 7f01f10aa700 1 -- 172.21.15.110:0/2486461218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01ec15a710 msgr2=0x7f01ec15d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.379+0000 7f01f10aa700 1 --2- 172.21.15.110:0/2486461218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01ec15a710 0x7f01ec15d060 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f01ec0d0030 tx=0x7f01e4009840 comp rx=0 tx=0).stop 2023-06-22T18:07:17.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.379+0000 7f01f10aa700 1 -- 172.21.15.110:0/2486461218 shutdown_connections 2023-06-22T18:07:17.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.379+0000 7f01f10aa700 1 --2- 172.21.15.110:0/2486461218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01ec15a710 0x7f01ec15d060 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.379+0000 7f01f10aa700 1 --2- 172.21.15.110:0/2486461218 >> v2:172.21.15.116:3300/0 conn(0x7f01ec0cee50 0x7f01ec15a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.379+0000 7f01f10aa700 1 -- 172.21.15.110:0/2486461218 >> 172.21.15.110:0/2486461218 conn(0x7f01ec0d0250 msgr2=0x7f01ec0d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.379+0000 7f01f10aa700 1 -- 172.21.15.110:0/2486461218 shutdown_connections 2023-06-22T18:07:17.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.379+0000 7f01f10aa700 1 -- 172.21.15.110:0/2486461218 wait complete. 2023-06-22T18:07:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.380+0000 7f01f10aa700 1 Processor -- start 2023-06-22T18:07:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.380+0000 7f01f10aa700 1 -- start start 2023-06-22T18:07:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01f10aa700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01ec06a080 0x7f01ec0b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.388 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01f10aa700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01ec0cee50 0x7f01ec155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01f10aa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01ec15a710 0x7f01ec0b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01f10aa700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0b2c00 con 0x7f01ec0cee50 2023-06-22T18:07:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01f10aa700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0c8480 con 0x7f01ec15a710 2023-06-22T18:07:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01f10aa700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec0b23e0 con 0x7f01ec06a080 2023-06-22T18:07:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01eb7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01ec0cee50 0x7f01ec155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01eb7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01ec0cee50 0x7f01ec155d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42558/0 (socket says 172.21.15.110:42558) 2023-06-22T18:07:17.389 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01f08a9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01ec15a710 0x7f01ec0b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:17.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01eb7fe700 1 -- 172.21.15.110:0/421532863 learned_addr learned my addr 172.21.15.110:0/421532863 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:17.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01ebfff700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01ec06a080 0x7f01ec0b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:17.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01eb7fe700 1 -- 172.21.15.110:0/421532863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01ec06a080 msgr2=0x7f01ec0b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01eb7fe700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01ec06a080 0x7f01ec0b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01eb7fe700 1 -- 172.21.15.110:0/421532863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01ec15a710 msgr2=0x7f01ec0b53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01eb7fe700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01ec15a710 0x7f01ec0b53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.390 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.381+0000 7f01eb7fe700 1 -- 172.21.15.110:0/421532863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e4009ae0 con 0x7f01ec0cee50 2023-06-22T18:07:17.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.382+0000 7f01eb7fe700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01ec0cee50 0x7f01ec155d40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f01e000b7e0 tx=0x7f01e0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:17.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.382+0000 7f01d3fff700 1 -- 172.21.15.110:0/421532863 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e000f510 con 0x7f01ec0cee50 2023-06-22T18:07:17.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.382+0000 7f01d3fff700 1 -- 172.21.15.110:0/421532863 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e000f670 con 0x7f01ec0cee50 2023-06-22T18:07:17.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.382+0000 7f01d3fff700 1 -- 172.21.15.110:0/421532863 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e000fa30 con 0x7f01ec0cee50 2023-06-22T18:07:17.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.382+0000 7f01f10aa700 1 -- 172.21.15.110:0/421532863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec0c7870 con 0x7f01ec0cee50 2023-06-22T18:07:17.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.382+0000 7f01f10aa700 1 -- 172.21.15.110:0/421532863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec0c7e30 con 0x7f01ec0cee50 2023-06-22T18:07:17.391 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.383+0000 7f01f10aa700 1 -- 172.21.15.110:0/421532863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ec070e80 con 0x7f01ec0cee50 2023-06-22T18:07:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.383+0000 7f01d3fff700 1 -- 172.21.15.110:0/421532863 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f01e001a400 con 0x7f01ec0cee50 2023-06-22T18:07:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.383+0000 7f01d3fff700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f01d8066e90 0x7f01d8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.383+0000 7f01d3fff700 1 -- 172.21.15.110:0/421532863 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5958+0+0 (secure 0 0 0) 0x7f01e0084000 con 0x7f01ec0cee50 2023-06-22T18:07:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.383+0000 7f01ebfff700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f01d8066e90 0x7f01d8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:17.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.384+0000 7f01ebfff700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f01d8066e90 0x7f01d8069340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f01dc006fd0 tx=0x7f01dc0090f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:17.393 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.388+0000 7f01d3fff700 1 -- 172.21.15.110:0/421532863 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f01e0051230 con 0x7f01ec0cee50 2023-06-22T18:07:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.615+0000 7f01f10aa700 1 -- 172.21.15.110:0/421532863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f01ec0b3bd0 con 0x7f01ec0cee50 2023-06-22T18:07:17.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.620+0000 7f01d3fff700 1 -- 172.21.15.110:0/421532863 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f01ec0b3bd0 con 0x7f01ec0cee50 2023-06-22T18:07:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.624+0000 7f01f10aa700 1 -- 172.21.15.110:0/421532863 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f01d8066e90 msgr2=0x7f01d8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.624+0000 7f01f10aa700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f01d8066e90 0x7f01d8069340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f01dc006fd0 tx=0x7f01dc0090f0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.624+0000 7f01f10aa700 1 -- 172.21.15.110:0/421532863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01ec0cee50 msgr2=0x7f01ec155d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.624+0000 7f01f10aa700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01ec0cee50 0x7f01ec155d40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f01e000b7e0 tx=0x7f01e0009220 comp rx=0 tx=0).stop 2023-06-22T18:07:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.627+0000 7f01f10aa700 1 -- 172.21.15.110:0/421532863 shutdown_connections 2023-06-22T18:07:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.627+0000 7f01f10aa700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01ec15a710 0x7f01ec0b53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.627+0000 7f01f10aa700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f01d8066e90 0x7f01d8069340 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.627+0000 7f01f10aa700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f01ec06a080 0x7f01ec0b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.627+0000 7f01f10aa700 1 --2- 172.21.15.110:0/421532863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01ec0cee50 0x7f01ec155d40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.627+0000 7f01f10aa700 1 -- 172.21.15.110:0/421532863 >> 172.21.15.110:0/421532863 conn(0x7f01ec0d0250 msgr2=0x7f01ec157ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.629+0000 7f01f10aa700 1 -- 172.21.15.110:0/421532863 shutdown_connections 2023-06-22T18:07:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.629+0000 7f01f10aa700 1 -- 172.21.15.110:0/421532863 wait complete. 2023-06-22T18:07:17.643 INFO:teuthology.orchestra.run.smithi110.stderr:new fs with metadata pool 3 and data pool 4 2023-06-22T18:07:17.655 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T18:07:17.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.765+0000 7f97fa719700 1 Processor -- start 2023-06-22T18:07:17.768 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.765+0000 7f97fa719700 1 -- start start 2023-06-22T18:07:17.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97fa719700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f97f4153880 0x7f97f40cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.770 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97fa719700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f97f40cd480 0x7f97f415a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97fa719700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f415d040 con 0x7f97f4153c50 2023-06-22T18:07:17.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97fa719700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f415aab0 con 0x7f97f4153880 2023-06-22T18:07:17.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97fa719700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f415e050 con 0x7f97f40cd480 2023-06-22T18:07:17.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f3fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f97f4153880 0x7f97f40cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:17.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f37fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f97f4153c50 0x7f97f40ccd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:42388/0 (socket says 172.21.15.110:42388) 2023-06-22T18:07:17.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f37fe700 1 -- 172.21.15.110:0/2327471699 learned_addr learned my addr 172.21.15.110:0/2327471699 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-06-22T18:07:17.771 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f8cb6700 1 --2- 172.21.15.110:0/2327471699 >> v2:172.21.15.116:3300/0 conn(0x7f97f40cd480 0x7f97f415a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:17.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f3fff700 1 -- 172.21.15.110:0/2327471699 >> v2:172.21.15.116:3300/0 conn(0x7f97f40cd480 msgr2=0x7f97f415a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:17.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f3fff700 1 --2- 172.21.15.110:0/2327471699 >> v2:172.21.15.116:3300/0 conn(0x7f97f40cd480 0x7f97f415a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f3fff700 1 -- 172.21.15.110:0/2327471699 >> v1:172.21.15.110:6789/0 conn(0x7f97f4153c50 legacy=0x7f97f40ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f3fff700 1 -- 172.21.15.110:0/2327471699 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f415dbc0 con 0x7f97f4153880 2023-06-22T18:07:17.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f3fff700 1 --2- 172.21.15.110:0/2327471699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f4153880 0x7f97f40cb3d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f97f4157610 tx=0x7f97e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a335e353c464c3 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:17.772 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f2ffd700 1 -- 172.21.15.110:0/2327471699 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8009e80 con 0x7f97f4153880 2023-06-22T18:07:17.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f2ffd700 1 -- 172.21.15.110:0/2327471699 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e800b8b0 con 0x7f97f4153880 2023-06-22T18:07:17.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.766+0000 7f97f2ffd700 1 -- 172.21.15.110:0/2327471699 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e800bc30 con 0x7f97f4153880 2023-06-22T18:07:17.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.767+0000 7f97fa719700 1 -- 172.21.15.110:0/2327471699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f4153880 msgr2=0x7f97f40cb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.767+0000 7f97fa719700 1 --2- 172.21.15.110:0/2327471699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f4153880 0x7f97f40cb3d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f97f4157610 tx=0x7f97e80091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.767+0000 7f97fa719700 1 -- 172.21.15.110:0/2327471699 shutdown_connections 2023-06-22T18:07:17.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.767+0000 7f97fa719700 1 --2- 172.21.15.110:0/2327471699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f4153880 0x7f97f40cb3d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.773 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.767+0000 7f97fa719700 1 --2- 172.21.15.110:0/2327471699 >> v2:172.21.15.116:3300/0 conn(0x7f97f40cd480 0x7f97f415a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.767+0000 7f97fa719700 1 -- 172.21.15.110:0/2327471699 >> 172.21.15.110:0/2327471699 conn(0x7f97f4067340 msgr2=0x7f97f4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:17.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.767+0000 7f97fa719700 1 -- 172.21.15.110:0/2327471699 shutdown_connections 2023-06-22T18:07:17.774 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.767+0000 7f97fa719700 1 -- 172.21.15.110:0/2327471699 wait complete. 2023-06-22T18:07:17.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.768+0000 7f97fa719700 1 Processor -- start 2023-06-22T18:07:17.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.768+0000 7f97fa719700 1 -- start start 2023-06-22T18:07:17.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97fa719700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f40cd480 0x7f97f4154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97fa719700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97f4153880 0x7f97f4151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.776 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97fa719700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97f4153c50 0x7f97f4157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97fa719700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f40c86e0 con 0x7f97f40cd480 2023-06-22T18:07:17.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97fa719700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f40c8da0 con 0x7f97f4153880 2023-06-22T18:07:17.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97fa719700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f4154840 con 0x7f97f4153c50 2023-06-22T18:07:17.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97f3fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f40cd480 0x7f97f4154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:17.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97f3fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f40cd480 0x7f97f4154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42572/0 (socket says 172.21.15.110:42572) 2023-06-22T18:07:17.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97f37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97f4153880 0x7f97f4151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:17.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97f3fff700 1 -- 172.21.15.110:0/2503328959 learned_addr learned my addr 172.21.15.110:0/2503328959 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:17.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97f8cb6700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97f4153c50 0x7f97f4157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:17.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97f3fff700 1 -- 172.21.15.110:0/2503328959 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97f4153c50 msgr2=0x7f97f4157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97f3fff700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97f4153c50 0x7f97f4157a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97f3fff700 1 -- 172.21.15.110:0/2503328959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97f4153880 msgr2=0x7f97f4151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97f3fff700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97f4153880 0x7f97f4151e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97f3fff700 1 -- 172.21.15.110:0/2503328959 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e8009420 con 0x7f97f40cd480 2023-06-22T18:07:17.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97f3fff700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f40cd480 0x7f97f4154300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f97e800b3c0 tx=0x7f97e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:17.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.769+0000 7f97dbfff700 1 -- 172.21.15.110:0/2503328959 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8011420 con 0x7f97f40cd480 2023-06-22T18:07:17.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.770+0000 7f97dbfff700 1 -- 172.21.15.110:0/2503328959 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e8011580 con 0x7f97f40cd480 2023-06-22T18:07:17.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.770+0000 7f97fa719700 1 -- 172.21.15.110:0/2503328959 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f41523b0 con 0x7f97f40cd480 2023-06-22T18:07:17.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.770+0000 7f97dbfff700 1 -- 172.21.15.110:0/2503328959 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8011940 con 0x7f97f40cd480 2023-06-22T18:07:17.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.770+0000 7f97fa719700 1 -- 172.21.15.110:0/2503328959 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f40c8400 con 0x7f97f40cd480 2023-06-22T18:07:17.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.771+0000 7f97fa719700 1 -- 172.21.15.110:0/2503328959 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f404c1b0 con 0x7f97f40cd480 2023-06-22T18:07:17.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.771+0000 7f97dbfff700 1 -- 172.21.15.110:0/2503328959 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f97e80193b0 con 0x7f97f40cd480 2023-06-22T18:07:17.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.771+0000 7f97dbfff700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f97e0066e90 0x7f97e0069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:17.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.772+0000 7f97f37fe700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f97e0066e90 0x7f97e0069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:17.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.772+0000 7f97dbfff700 1 -- 172.21.15.110:0/2503328959 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f97e8086290 con 0x7f97f40cd480 2023-06-22T18:07:17.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.772+0000 7f97f37fe700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f97e0066e90 0x7f97e0069340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f97f4159030 tx=0x7f97e4000bc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:17.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.776+0000 7f97dbfff700 1 -- 172.21.15.110:0/2503328959 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f97e80533c0 con 0x7f97f40cd480 2023-06-22T18:07:17.963 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.960+0000 7f97fa719700 1 -- 172.21.15.110:0/2503328959 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97f40ca7b0 con 0x7f97f40cd480 2023-06-22T18:07:17.964 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.961+0000 7f97dbfff700 1 -- 172.21.15.110:0/2503328959 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17023 (secure 0 0 0) 0x7f97e80569e0 con 0x7f97f40cd480 2023-06-22T18:07:17.965 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:07:17.965 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":20,"fsid":"cbed9da7-33df-4858-8860-5c7cacb062c0","created":"2023-06-22T18:06:59.330561+0000","modified":"2023-06-22T18:07:17.616489+0000","last_up_change":"2023-06-22T18:07:09.187638+0000","last_in_change":"2023-06-22T18:07:01.731974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-22T18:07:10.484545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-22T18:07:10.898910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T18:07:15.279292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T18:07:16.590008+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66c18197-0807-4f09-9e82-decc44f8bb6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":117902},{"type":"v1","addr":"172.21.15.110:6801","nonce":117902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":117902},{"type":"v1","addr":"172.21.15.110:6805","nonce":117902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":117902},{"type":"v1","addr":"172.21.15.110:6813","nonce":117902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":117902},{"type":"v1","addr":"172.21.15.110:6809","nonce":117902}]},"public_addr":"172.21.15.110:6801/117902","cluster_addr":"172.21.15.110:6805/117902","heartbeat_back_addr":"172.21.15.110:6813/117902","heartbeat_front_addr":"172.21.15.110:6809/117902","state":["exists","up"]},{"osd":1,"uuid":"e2e2e368-b287-4ae9-a502-f2d0276de124","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":117903},{"type":"v1","addr":"172.21.15.110:6804","nonce":117903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":117903},{"type":"v1","addr":"172.21.15.110:6808","nonce":117903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":117903},{"type":"v1","addr":"172.21.15.110:6815","nonce":117903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":117903},{"type":"v1","addr":"172.21.15.110:6812","nonce":117903}]},"public_addr":"172.21.15.110:6804/117903","cluster_addr":"172.21.15.110:6808/117903","heartbeat_back_addr":"172.21.15.110:6815/117903","heartbeat_front_addr":"172.21.15.110:6812/117903","state":["exists","up"]},{"osd":2,"uuid":"89608384-c8bd-4032-98e6-6dd77f29f4de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":117932},{"type":"v1","addr":"172.21.15.110:6819","nonce":117932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6821","nonce":117932},{"type":"v1","addr":"172.21.15.110:6823","nonce":117932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6829","nonce":117932},{"type":"v1","addr":"172.21.15.110:6831","nonce":117932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6825","nonce":117932},{"type":"v1","addr":"172.21.15.110:6827","nonce":117932}]},"public_addr":"172.21.15.110:6819/117932","cluster_addr":"172.21.15.110:6823/117932","heartbeat_back_addr":"172.21.15.110:6831/117932","heartbeat_front_addr":"172.21.15.110:6827/117932","state":["exists","up"]},{"osd":3,"uuid":"e20af00e-ab57-4686-87b0-a60c90128569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":117933},{"type":"v1","addr":"172.21.15.110:6818","nonce":117933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":117933},{"type":"v1","addr":"172.21.15.110:6822","nonce":117933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":117933},{"type":"v1","addr":"172.21.15.110:6830","nonce":117933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":117933},{"type":"v1","addr":"172.21.15.110:6826","nonce":117933}]},"public_addr":"172.21.15.110:6818/117933","cluster_addr":"172.21.15.110:6822/117933","heartbeat_back_addr":"172.21.15.110:6830/117933","heartbeat_front_addr":"172.21.15.110:6826/117933","state":["exists","up"]},{"osd":4,"uuid":"a9142dbb-15ab-43a0-8501-5ca261a415ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":118295},{"type":"v1","addr":"172.21.15.116:6801","nonce":118295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":118295},{"type":"v1","addr":"172.21.15.116:6803","nonce":118295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":118295},{"type":"v1","addr":"172.21.15.116:6807","nonce":118295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":118295},{"type":"v1","addr":"172.21.15.116:6805","nonce":118295}]},"public_addr":"172.21.15.116:6801/118295","cluster_addr":"172.21.15.116:6803/118295","heartbeat_back_addr":"172.21.15.116:6807/118295","heartbeat_front_addr":"172.21.15.116:6805/118295","state":["exists","up"]},{"osd":5,"uuid":"e81f0aa0-d55d-4d72-937e-a1bc82b3117f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":118312},{"type":"v1","addr":"172.21.15.116:6825","nonce":118312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":118312},{"type":"v1","addr":"172.21.15.116:6827","nonce":118312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":118312},{"type":"v1","addr":"172.21.15.116:6831","nonce":118312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":118312},{"type":"v1","addr":"172.21.15.116:6829","nonce":118312}]},"public_addr":"172.21.15.116:6825/118312","cluster_addr":"172.21.15.116:6827/118312","heartbeat_back_addr":"172.21.15.116:6831/118312","heartbeat_front_addr":"172.21.15.116:6829/118312","state":["exists","up"]},{"osd":6,"uuid":"25f64c11-e6e8-46d1-b064-28743c4233df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":118296},{"type":"v1","addr":"172.21.15.116:6810","nonce":118296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":118296},{"type":"v1","addr":"172.21.15.116:6814","nonce":118296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":118296},{"type":"v1","addr":"172.21.15.116:6822","nonce":118296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":118296},{"type":"v1","addr":"172.21.15.116:6818","nonce":118296}]},"public_addr":"172.21.15.116:6810/118296","cluster_addr":"172.21.15.116:6814/118296","heartbeat_back_addr":"172.21.15.116:6822/118296","heartbeat_front_addr":"172.21.15.116:6818/118296","state":["exists","up"]},{"osd":7,"uuid":"608edcd6-65d3-4e76-bbd4-a612a6befa4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":118297},{"type":"v1","addr":"172.21.15.116:6811","nonce":118297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":118297},{"type":"v1","addr":"172.21.15.116:6815","nonce":118297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":118297},{"type":"v1","addr":"172.21.15.116:6823","nonce":118297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":118297},{"type":"v1","addr":"172.21.15.116:6819","nonce":118297}]},"public_addr":"172.21.15.116:6811/118297","cluster_addr":"172.21.15.116:6815/118297","heartbeat_back_addr":"172.21.15.116:6823/118297","heartbeat_front_addr":"172.21.15.116:6819/118297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.824390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.872398+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.931009+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.875583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.632902+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.874695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.702422+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.626714+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T18:07:17.966 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 -- 172.21.15.110:0/2503328959 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f97e0066e90 msgr2=0x7f97e0069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f97e0066e90 0x7f97e0069340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f97f4159030 tx=0x7f97e4000bc0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 -- 172.21.15.110:0/2503328959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f40cd480 msgr2=0x7f97f4154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:17.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f40cd480 0x7f97f4154300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f97e800b3c0 tx=0x7f97e8017000 comp rx=0 tx=0).stop 2023-06-22T18:07:17.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 -- 172.21.15.110:0/2503328959 shutdown_connections 2023-06-22T18:07:17.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f97e0066e90 0x7f97e0069340 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f97f4153c50 0x7f97f4157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.967 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f40cd480 0x7f97f4154300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 --2- 172.21.15.110:0/2503328959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97f4153880 0x7f97f4151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 -- 172.21.15.110:0/2503328959 >> 172.21.15.110:0/2503328959 conn(0x7f97f4067340 msgr2=0x7f97f4067710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 -- 172.21.15.110:0/2503328959 shutdown_connections 2023-06-22T18:07:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:17.963+0000 7f97fa719700 1 -- 172.21.15.110:0/2503328959 wait complete. 2023-06-22T18:07:17.978 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-22T18:07:18.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.088+0000 7fe88bfff700 1 Processor -- start 2023-06-22T18:07:18.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.088+0000 7fe88bfff700 1 -- start start 2023-06-22T18:07:18.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88bfff700 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe88c159060 0x7fe88c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88bfff700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe88c067a50 0x7fe88c0ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88bfff700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88c0cfe00 con 0x7fe88c15ce40 2023-06-22T18:07:18.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88bfff700 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88c0afbd0 con 0x7fe88c159060 2023-06-22T18:07:18.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88bfff700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c0cb7a0 con 0x7fe88c067a50 2023-06-22T18:07:18.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88a7fc700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe88c15ce40 0x7fe88c067340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:42390/0 (socket says 172.21.15.110:42390) 2023-06-22T18:07:18.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88a7fc700 1 -- 172.21.15.110:0/3731814209 learned_addr learned my addr 172.21.15.110:0/3731814209 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-06-22T18:07:18.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88affd700 1 --2- 172.21.15.110:0/3731814209 >> v2:172.21.15.116:3300/0 conn(0x7fe88c159060 0x7fe88c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88b7fe700 1 --2- 172.21.15.110:0/3731814209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe88c067a50 0x7fe88c0ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe889ffb700 1 -- 172.21.15.110:0/3731814209 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549505800 0 0) 0x7fe88c0cfe00 con 0x7fe88c15ce40 2023-06-22T18:07:18.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe889ffb700 1 -- 172.21.15.110:0/3731814209 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe874003570 con 0x7fe88c15ce40 2023-06-22T18:07:18.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88affd700 1 -- 172.21.15.110:0/3731814209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe88c067a50 msgr2=0x7fe88c0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:18.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88affd700 1 --2- 172.21.15.110:0/3731814209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe88c067a50 0x7fe88c0ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88affd700 1 -- 172.21.15.110:0/3731814209 >> v1:172.21.15.110:6789/0 conn(0x7fe88c15ce40 legacy=0x7fe88c067340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe889ffb700 1 -- 172.21.15.110:0/3731814209 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1694628288 0 0) 0x7fe874003570 con 0x7fe88c15ce40 2023-06-22T18:07:18.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.089+0000 7fe88affd700 1 -- 172.21.15.110:0/3731814209 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88c15d320 con 0x7fe88c159060 2023-06-22T18:07:18.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe88affd700 1 --2- 172.21.15.110:0/3731814209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88c159060 0x7fe88c0cb260 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009e60 tx=0x7fe87c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87429953adb862ec server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:18.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe889ffb700 1 -- 172.21.15.110:0/3731814209 <== mon.1 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00e070 con 0x7fe88c159060 2023-06-22T18:07:18.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe889ffb700 1 -- 172.21.15.110:0/3731814209 <== mon.1 v2:172.21.15.116:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c00b7a0 con 0x7fe88c159060 2023-06-22T18:07:18.096 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe889ffb700 1 -- 172.21.15.110:0/3731814209 <== mon.1 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880001470 con 0x7fe88c159060 2023-06-22T18:07:18.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe88bfff700 1 -- 172.21.15.110:0/3731814209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88c159060 msgr2=0x7fe88c0cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe88bfff700 1 --2- 172.21.15.110:0/3731814209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88c159060 0x7fe88c0cb260 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009e60 tx=0x7fe87c009780 comp rx=0 tx=0).stop 2023-06-22T18:07:18.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe88bfff700 1 -- 172.21.15.110:0/3731814209 shutdown_connections 2023-06-22T18:07:18.098 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe88bfff700 1 --2- 172.21.15.110:0/3731814209 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe88c067a50 0x7fe88c0ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe88bfff700 1 --2- 172.21.15.110:0/3731814209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88c159060 0x7fe88c0cb260 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe88bfff700 1 -- 172.21.15.110:0/3731814209 >> 172.21.15.110:0/3731814209 conn(0x7fe88c0ccc80 msgr2=0x7fe88c0cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:18.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe88bfff700 1 -- 172.21.15.110:0/3731814209 shutdown_connections 2023-06-22T18:07:18.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.090+0000 7fe88bfff700 1 -- 172.21.15.110:0/3731814209 wait complete. 2023-06-22T18:07:18.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.092+0000 7fe88bfff700 1 Processor -- start 2023-06-22T18:07:18.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.092+0000 7fe88bfff700 1 -- start start 2023-06-22T18:07:18.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.092+0000 7fe88bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88c067a50 0x7fe88c0b5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.099 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.092+0000 7fe88bfff700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe88c159060 0x7fe88c0b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.092+0000 7fe88bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe88c15ce40 0x7fe88c1536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.092+0000 7fe88bfff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c0b5f60 con 0x7fe88c15ce40 2023-06-22T18:07:18.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.092+0000 7fe88bfff700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c0b4450 con 0x7fe88c067a50 2023-06-22T18:07:18.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.092+0000 7fe88bfff700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c0b3a80 con 0x7fe88c159060 2023-06-22T18:07:18.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe88c15ce40 0x7fe88c1536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe88c15ce40 0x7fe88c1536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42580/0 (socket says 172.21.15.110:42580) 2023-06-22T18:07:18.100 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88b7fe700 1 -- 172.21.15.110:0/1495044187 learned_addr learned my addr 172.21.15.110:0/1495044187 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:18.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88a7fc700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe88c159060 0x7fe88c0b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88affd700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88c067a50 0x7fe88c0b5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88b7fe700 1 -- 172.21.15.110:0/1495044187 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe88c159060 msgr2=0x7fe88c0b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88b7fe700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe88c159060 0x7fe88c0b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88b7fe700 1 -- 172.21.15.110:0/1495044187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88c067a50 msgr2=0x7fe88c0b5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88b7fe700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88c067a50 0x7fe88c0b5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.101 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88b7fe700 1 -- 172.21.15.110:0/1495044187 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c009a20 con 0x7fe88c15ce40 2023-06-22T18:07:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88b7fe700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe88c15ce40 0x7fe88c1536e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe88400ea50 tx=0x7fe8840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe872ffd700 1 -- 172.21.15.110:0/1495044187 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88400c8c0 con 0x7fe88c15ce40 2023-06-22T18:07:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe872ffd700 1 -- 172.21.15.110:0/1495044187 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88400ca40 con 0x7fe88c15ce40 2023-06-22T18:07:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe872ffd700 1 -- 172.21.15.110:0/1495044187 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88400ce00 con 0x7fe88c15ce40 2023-06-22T18:07:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88bfff700 1 -- 172.21.15.110:0/1495044187 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88c154280 con 0x7fe88c15ce40 2023-06-22T18:07:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.093+0000 7fe88bfff700 1 -- 172.21.15.110:0/1495044187 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88c0b3430 con 0x7fe88c15ce40 2023-06-22T18:07:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.094+0000 7fe88bfff700 1 -- 172.21.15.110:0/1495044187 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88c04c1b0 con 0x7fe88c15ce40 2023-06-22T18:07:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.094+0000 7fe872ffd700 1 -- 172.21.15.110:0/1495044187 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fe884013680 con 0x7fe88c15ce40 2023-06-22T18:07:18.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.095+0000 7fe872ffd700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe878066e90 0x7fe878069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.095+0000 7fe872ffd700 1 -- 172.21.15.110:0/1495044187 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe884084940 con 0x7fe88c15ce40 2023-06-22T18:07:18.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.095+0000 7fe88affd700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe878066e90 0x7fe878069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.103 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.095+0000 7fe88affd700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe878066e90 0x7fe878069340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0097c0 tx=0x7fe87c018000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:18.104 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.099+0000 7fe872ffd700 1 -- 172.21.15.110:0/1495044187 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fe884051b10 con 0x7fe88c15ce40 2023-06-22T18:07:18.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.284+0000 7fe88bfff700 1 -- 172.21.15.110:0/1495044187 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe88c0b3bc0 con 0x7fe88c15ce40 2023-06-22T18:07:18.288 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.285+0000 7fe872ffd700 1 -- 172.21.15.110:0/1495044187 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+17023 (secure 0 0 0) 0x7fe884055130 con 0x7fe88c15ce40 2023-06-22T18:07:18.288 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-06-22T18:07:18.289 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":20,"fsid":"cbed9da7-33df-4858-8860-5c7cacb062c0","created":"2023-06-22T18:06:59.330561+0000","modified":"2023-06-22T18:07:17.616489+0000","last_up_change":"2023-06-22T18:07:09.187638+0000","last_in_change":"2023-06-22T18:07:01.731974+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-22T18:07:10.484545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-22T18:07:10.898910+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-22T18:07:15.279292+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-22T18:07:16.590008+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"66c18197-0807-4f09-9e82-decc44f8bb6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":117902},{"type":"v1","addr":"172.21.15.110:6801","nonce":117902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":117902},{"type":"v1","addr":"172.21.15.110:6805","nonce":117902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":117902},{"type":"v1","addr":"172.21.15.110:6813","nonce":117902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":117902},{"type":"v1","addr":"172.21.15.110:6809","nonce":117902}]},"public_addr":"172.21.15.110:6801/117902","cluster_addr":"172.21.15.110:6805/117902","heartbeat_back_addr":"172.21.15.110:6813/117902","heartbeat_front_addr":"172.21.15.110:6809/117902","state":["exists","up"]},{"osd":1,"uuid":"e2e2e368-b287-4ae9-a502-f2d0276de124","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":117903},{"type":"v1","addr":"172.21.15.110:6804","nonce":117903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":117903},{"type":"v1","addr":"172.21.15.110:6808","nonce":117903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":117903},{"type":"v1","addr":"172.21.15.110:6815","nonce":117903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":117903},{"type":"v1","addr":"172.21.15.110:6812","nonce":117903}]},"public_addr":"172.21.15.110:6804/117903","cluster_addr":"172.21.15.110:6808/117903","heartbeat_back_addr":"172.21.15.110:6815/117903","heartbeat_front_addr":"172.21.15.110:6812/117903","state":["exists","up"]},{"osd":2,"uuid":"89608384-c8bd-4032-98e6-6dd77f29f4de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6817","nonce":117932},{"type":"v1","addr":"172.21.15.110:6819","nonce":117932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6821","nonce":117932},{"type":"v1","addr":"172.21.15.110:6823","nonce":117932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6829","nonce":117932},{"type":"v1","addr":"172.21.15.110:6831","nonce":117932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6825","nonce":117932},{"type":"v1","addr":"172.21.15.110:6827","nonce":117932}]},"public_addr":"172.21.15.110:6819/117932","cluster_addr":"172.21.15.110:6823/117932","heartbeat_back_addr":"172.21.15.110:6831/117932","heartbeat_front_addr":"172.21.15.110:6827/117932","state":["exists","up"]},{"osd":3,"uuid":"e20af00e-ab57-4686-87b0-a60c90128569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":117933},{"type":"v1","addr":"172.21.15.110:6818","nonce":117933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":117933},{"type":"v1","addr":"172.21.15.110:6822","nonce":117933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":117933},{"type":"v1","addr":"172.21.15.110:6830","nonce":117933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":117933},{"type":"v1","addr":"172.21.15.110:6826","nonce":117933}]},"public_addr":"172.21.15.110:6818/117933","cluster_addr":"172.21.15.110:6822/117933","heartbeat_back_addr":"172.21.15.110:6830/117933","heartbeat_front_addr":"172.21.15.110:6826/117933","state":["exists","up"]},{"osd":4,"uuid":"a9142dbb-15ab-43a0-8501-5ca261a415ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":118295},{"type":"v1","addr":"172.21.15.116:6801","nonce":118295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":118295},{"type":"v1","addr":"172.21.15.116:6803","nonce":118295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":118295},{"type":"v1","addr":"172.21.15.116:6807","nonce":118295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":118295},{"type":"v1","addr":"172.21.15.116:6805","nonce":118295}]},"public_addr":"172.21.15.116:6801/118295","cluster_addr":"172.21.15.116:6803/118295","heartbeat_back_addr":"172.21.15.116:6807/118295","heartbeat_front_addr":"172.21.15.116:6805/118295","state":["exists","up"]},{"osd":5,"uuid":"e81f0aa0-d55d-4d72-937e-a1bc82b3117f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":118312},{"type":"v1","addr":"172.21.15.116:6825","nonce":118312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":118312},{"type":"v1","addr":"172.21.15.116:6827","nonce":118312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":118312},{"type":"v1","addr":"172.21.15.116:6831","nonce":118312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":118312},{"type":"v1","addr":"172.21.15.116:6829","nonce":118312}]},"public_addr":"172.21.15.116:6825/118312","cluster_addr":"172.21.15.116:6827/118312","heartbeat_back_addr":"172.21.15.116:6831/118312","heartbeat_front_addr":"172.21.15.116:6829/118312","state":["exists","up"]},{"osd":6,"uuid":"25f64c11-e6e8-46d1-b064-28743c4233df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":118296},{"type":"v1","addr":"172.21.15.116:6810","nonce":118296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":118296},{"type":"v1","addr":"172.21.15.116:6814","nonce":118296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":118296},{"type":"v1","addr":"172.21.15.116:6822","nonce":118296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":118296},{"type":"v1","addr":"172.21.15.116:6818","nonce":118296}]},"public_addr":"172.21.15.116:6810/118296","cluster_addr":"172.21.15.116:6814/118296","heartbeat_back_addr":"172.21.15.116:6822/118296","heartbeat_front_addr":"172.21.15.116:6818/118296","state":["exists","up"]},{"osd":7,"uuid":"608edcd6-65d3-4e76-bbd4-a612a6befa4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6809","nonce":118297},{"type":"v1","addr":"172.21.15.116:6811","nonce":118297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6813","nonce":118297},{"type":"v1","addr":"172.21.15.116:6815","nonce":118297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6821","nonce":118297},{"type":"v1","addr":"172.21.15.116:6823","nonce":118297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6817","nonce":118297},{"type":"v1","addr":"172.21.15.116:6819","nonce":118297}]},"public_addr":"172.21.15.116:6811/118297","cluster_addr":"172.21.15.116:6815/118297","heartbeat_back_addr":"172.21.15.116:6823/118297","heartbeat_front_addr":"172.21.15.116:6819/118297","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.824390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.872398+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.931009+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.875583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.632902+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.874695+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.702422+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-22T18:07:06.626714+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-22T18:07:18.290 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 -- 172.21.15.110:0/1495044187 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe878066e90 msgr2=0x7fe878069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe878066e90 0x7fe878069340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0097c0 tx=0x7fe87c018000 comp rx=0 tx=0).stop 2023-06-22T18:07:18.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 -- 172.21.15.110:0/1495044187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe88c15ce40 msgr2=0x7fe88c1536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe88c15ce40 0x7fe88c1536e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe88400ea50 tx=0x7fe8840091a0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 -- 172.21.15.110:0/1495044187 shutdown_connections 2023-06-22T18:07:18.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88c067a50 0x7fe88c0b5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.291 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7fe878066e90 0x7fe878069340 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.292 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe88c15ce40 0x7fe88c1536e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.292 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 --2- 172.21.15.110:0/1495044187 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7fe88c159060 0x7fe88c0b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.292 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 -- 172.21.15.110:0/1495044187 >> 172.21.15.110:0/1495044187 conn(0x7fe88c0ccc80 msgr2=0x7fe88c158240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:18.292 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 -- 172.21.15.110:0/1495044187 shutdown_connections 2023-06-22T18:07:18.292 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.287+0000 7fe88bfff700 1 -- 172.21.15.110:0/1495044187 wait complete. 2023-06-22T18:07:18.303 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-22T18:07:18.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.416+0000 7f8612a5a700 1 Processor -- start 2023-06-22T18:07:18.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.416+0000 7f8612a5a700 1 -- start start 2023-06-22T18:07:18.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.417+0000 7f8612a5a700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0d0380 0x7f860c0d0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.417+0000 7f8612a5a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f860c15da90 0x7f860c158f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.417+0000 7f8612a5a700 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860c0cf730 con 0x7f860c1594c0 2023-06-22T18:07:18.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.417+0000 7f8612a5a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860c0cfbc0 con 0x7f860c15da90 2023-06-22T18:07:18.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.417+0000 7f8612a5a700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c159940 con 0x7f860c0d0380 2023-06-22T18:07:18.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.417+0000 7f860b7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f860c15da90 0x7f860c158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.417+0000 7f860b7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f860c15da90 0x7f860c158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42592/0 (socket says 172.21.15.110:42592) 2023-06-22T18:07:18.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.417+0000 7f860b7fe700 1 -- 172.21.15.110:0/2968700062 learned_addr learned my addr 172.21.15.110:0/2968700062 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:18.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.418+0000 7f860b7fe700 1 -- 172.21.15.110:0/2968700062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0d0380 msgr2=0x7f860c0d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:18.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.418+0000 7f860bfff700 1 --2- 172.21.15.110:0/2968700062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0d0380 0x7f860c0d0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.418+0000 7f860b7fe700 1 --2- 172.21.15.110:0/2968700062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0d0380 0x7f860c0d0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.418+0000 7f860b7fe700 1 -- 172.21.15.110:0/2968700062 >> v1:172.21.15.116:6789/0 conn(0x7f860c1594c0 legacy=0x7f860c068a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:18.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.418+0000 7f860b7fe700 1 -- 172.21.15.110:0/2968700062 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c0ceb20 con 0x7f860c15da90 2023-06-22T18:07:18.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.418+0000 7f860b7fe700 1 --2- 172.21.15.110:0/2968700062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f860c15da90 0x7f860c158f80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f86000099b0 tx=0x7f8600009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ab61458a510885 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:18.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.418+0000 7f860affd700 1 -- 172.21.15.110:0/2968700062 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860000b820 con 0x7f860c15da90 2023-06-22T18:07:18.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.418+0000 7f860affd700 1 -- 172.21.15.110:0/2968700062 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860000b980 con 0x7f860c15da90 2023-06-22T18:07:18.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.418+0000 7f860affd700 1 -- 172.21.15.110:0/2968700062 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860000bd20 con 0x7f860c15da90 2023-06-22T18:07:18.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.419+0000 7f8612a5a700 1 -- 172.21.15.110:0/2968700062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f860c15da90 msgr2=0x7f860c158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.419+0000 7f8612a5a700 1 --2- 172.21.15.110:0/2968700062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f860c15da90 0x7f860c158f80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f86000099b0 tx=0x7f8600009250 comp rx=0 tx=0).stop 2023-06-22T18:07:18.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.419+0000 7f8612a5a700 1 -- 172.21.15.110:0/2968700062 shutdown_connections 2023-06-22T18:07:18.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.419+0000 7f8612a5a700 1 --2- 172.21.15.110:0/2968700062 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0d0380 0x7f860c0d0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.419+0000 7f8612a5a700 1 --2- 172.21.15.110:0/2968700062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f860c15da90 0x7f860c158f80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.419+0000 7f8612a5a700 1 -- 172.21.15.110:0/2968700062 >> 172.21.15.110:0/2968700062 conn(0x7f860c069f80 msgr2=0x7f860c06a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:18.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.419+0000 7f8612a5a700 1 -- 172.21.15.110:0/2968700062 shutdown_connections 2023-06-22T18:07:18.427 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.419+0000 7f8612a5a700 1 -- 172.21.15.110:0/2968700062 wait complete. 2023-06-22T18:07:18.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.420+0000 7f8612a5a700 1 Processor -- start 2023-06-22T18:07:18.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.420+0000 7f8612a5a700 1 -- start start 2023-06-22T18:07:18.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f8612a5a700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0d0380 0x7f860c155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f8612a5a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f860c1594c0 0x7f860c0b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f8612a5a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860c15da90 0x7f860c0b4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f8612a5a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0cc110 con 0x7f860c1594c0 2023-06-22T18:07:18.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f8612a5a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0c6110 con 0x7f860c15da90 2023-06-22T18:07:18.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f8612a5a700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0c5c80 con 0x7f860c0d0380 2023-06-22T18:07:18.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f860b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f860c1594c0 0x7f860c0b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f860b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f860c1594c0 0x7f860c0b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42602/0 (socket says 172.21.15.110:42602) 2023-06-22T18:07:18.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f860b7fe700 1 -- 172.21.15.110:0/3250958004 learned_addr learned my addr 172.21.15.110:0/3250958004 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:18.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f860bfff700 1 --2- 172.21.15.110:0/3250958004 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0d0380 0x7f860c155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f8610ff7700 1 --2- 172.21.15.110:0/3250958004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860c15da90 0x7f860c0b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f860b7fe700 1 -- 172.21.15.110:0/3250958004 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0d0380 msgr2=0x7f860c155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f860b7fe700 1 --2- 172.21.15.110:0/3250958004 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f860c0d0380 0x7f860c155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f860b7fe700 1 -- 172.21.15.110:0/3250958004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860c15da90 msgr2=0x7f860c0b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f860b7fe700 1 --2- 172.21.15.110:0/3250958004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f860c15da90 0x7f860c0b4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f860b7fe700 1 -- 172.21.15.110:0/3250958004 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86000094f0 con 0x7f860c1594c0 2023-06-22T18:07:18.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f860b7fe700 1 --2- 172.21.15.110:0/3250958004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f860c1594c0 0x7f860c0b3c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f86000094c0 tx=0x7f8600016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:18.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f85f3fff700 1 -- 172.21.15.110:0/3250958004 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860001c460 con 0x7f860c1594c0 2023-06-22T18:07:18.435 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f85f3fff700 1 -- 172.21.15.110:0/3250958004 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860001c5e0 con 0x7f860c1594c0 2023-06-22T18:07:18.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f8612a5a700 1 -- 172.21.15.110:0/3250958004 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860c156520 con 0x7f860c1594c0 2023-06-22T18:07:18.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f85f3fff700 1 -- 172.21.15.110:0/3250958004 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860001c9a0 con 0x7f860c1594c0 2023-06-22T18:07:18.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.421+0000 7f8612a5a700 1 -- 172.21.15.110:0/3250958004 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860c0c5630 con 0x7f860c1594c0 2023-06-22T18:07:18.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.422+0000 7f8612a5a700 1 -- 172.21.15.110:0/3250958004 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860c04c1b0 con 0x7f860c1594c0 2023-06-22T18:07:18.436 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.423+0000 7f85f3fff700 1 -- 172.21.15.110:0/3250958004 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f860001cd50 con 0x7f860c1594c0 2023-06-22T18:07:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.423+0000 7f85f3fff700 1 --2- 172.21.15.110:0/3250958004 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f85f8066e90 0x7f85f8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.423+0000 7f85f3fff700 1 -- 172.21.15.110:0/3250958004 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f8600086b80 con 0x7f860c1594c0 2023-06-22T18:07:18.437 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.423+0000 7f860bfff700 1 --2- 172.21.15.110:0/3250958004 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f85f8066e90 0x7f85f8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.423+0000 7f860bfff700 1 --2- 172.21.15.110:0/3250958004 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f85f8066e90 0x7f85f8069340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00ab30 tx=0x7f85fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:18.438 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.427+0000 7f85f3fff700 1 -- 172.21.15.110:0/3250958004 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f8600053d50 con 0x7f860c1594c0 2023-06-22T18:07:18.659 INFO:teuthology.orchestra.run.smithi110.stderr:no valid command found; 1 closest matches: 2023-06-22T18:07:18.660 INFO:teuthology.orchestra.run.smithi110.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-22T18:07:18.660 INFO:teuthology.orchestra.run.smithi110.stderr:Error EINVAL: invalid command 2023-06-22T18:07:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-22T18:07:18.663 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-22T18:07:18.663 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-22T18:07:18.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.778+0000 7f088dc86700 1 Processor -- start 2023-06-22T18:07:18.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.778+0000 7f088dc86700 1 -- start start 2023-06-22T18:07:18.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f088dc86700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f088806a080 0x7f08880ca890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f088dc86700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08880cee50 0x7f088815a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f088dc86700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0888159a60 con 0x7f088815a710 2023-06-22T18:07:18.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f088dc86700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08880afac0 con 0x7f08880cee50 2023-06-22T18:07:18.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f088dc86700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08880cadd0 con 0x7f088806a080 2023-06-22T18:07:18.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f087ffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08880cee50 0x7f088815a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f087ffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08880cee50 0x7f088815a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53088/0 (socket says 172.21.15.110:53088) 2023-06-22T18:07:18.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f087ffff700 1 -- 172.21.15.110:0/2330157559 learned_addr learned my addr 172.21.15.110:0/2330157559 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:18.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f088d485700 1 --1- 172.21.15.110:0/2330157559 >> v1:172.21.15.110:6789/0 conn(0x7f088815a710 0x7f088815d060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:48062/0 (socket says 172.21.15.110:48062) 2023-06-22T18:07:18.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f088cc84700 1 --2- 172.21.15.110:0/2330157559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f088806a080 0x7f08880ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f087ffff700 1 -- 172.21.15.110:0/2330157559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f088806a080 msgr2=0x7f08880ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-22T18:07:18.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f087ffff700 1 --2- 172.21.15.110:0/2330157559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f088806a080 0x7f08880ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f087ffff700 1 -- 172.21.15.110:0/2330157559 >> v1:172.21.15.110:6789/0 conn(0x7f088815a710 legacy=0x7f088815d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f087ffff700 1 -- 172.21.15.110:0/2330157559 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08880cb1d0 con 0x7f08880cee50 2023-06-22T18:07:18.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f087f7fe700 1 -- 172.21.15.110:0/2330157559 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434787492 0 0) 0x7f0888159a60 con 0x7f088815a710 2023-06-22T18:07:18.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.779+0000 7f087ffff700 1 --2- 172.21.15.110:0/2330157559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08880cee50 0x7f088815a1d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f08780099b0 tx=0x7f0878009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba30b31e3dbf2e7e server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:18.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.780+0000 7f087f7fe700 1 -- 172.21.15.110:0/2330157559 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087800b860 con 0x7f08880cee50 2023-06-22T18:07:18.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.780+0000 7f087f7fe700 1 -- 172.21.15.110:0/2330157559 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087800b9c0 con 0x7f08880cee50 2023-06-22T18:07:18.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.780+0000 7f087f7fe700 1 -- 172.21.15.110:0/2330157559 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087800bd60 con 0x7f08880cee50 2023-06-22T18:07:18.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.780+0000 7f088dc86700 1 -- 172.21.15.110:0/2330157559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08880cee50 msgr2=0x7f088815a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.780+0000 7f088dc86700 1 --2- 172.21.15.110:0/2330157559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08880cee50 0x7f088815a1d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f08780099b0 tx=0x7f0878009250 comp rx=0 tx=0).stop 2023-06-22T18:07:18.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.780+0000 7f088dc86700 1 -- 172.21.15.110:0/2330157559 shutdown_connections 2023-06-22T18:07:18.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.780+0000 7f088dc86700 1 --2- 172.21.15.110:0/2330157559 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f088806a080 0x7f08880ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.780+0000 7f088dc86700 1 --2- 172.21.15.110:0/2330157559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08880cee50 0x7f088815a1d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.780+0000 7f088dc86700 1 -- 172.21.15.110:0/2330157559 >> 172.21.15.110:0/2330157559 conn(0x7f08880d0250 msgr2=0x7f08880d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-22T18:07:18.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.780+0000 7f088dc86700 1 -- 172.21.15.110:0/2330157559 shutdown_connections 2023-06-22T18:07:18.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.780+0000 7f088dc86700 1 -- 172.21.15.110:0/2330157559 wait complete. 2023-06-22T18:07:18.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.781+0000 7f088dc86700 1 Processor -- start 2023-06-22T18:07:18.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.781+0000 7f088dc86700 1 -- start start 2023-06-22T18:07:18.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f088dc86700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088806a080 0x7f0888169870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f088dc86700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08880cee50 0x7f088815e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f088dc86700 1 --2- >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f088815a710 0x7f08880c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f088dc86700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08880c8090 con 0x7f08880cee50 2023-06-22T18:07:18.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f088dc86700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08880c7480 con 0x7f088806a080 2023-06-22T18:07:18.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f088dc86700 1 -- --> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08880c7760 con 0x7f088815a710 2023-06-22T18:07:18.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f087ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08880cee50 0x7f088815e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f087ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08880cee50 0x7f088815e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53102/0 (socket says 172.21.15.110:53102) 2023-06-22T18:07:18.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f087ffff700 1 -- 172.21.15.110:0/970673675 learned_addr learned my addr 172.21.15.110:0/970673675 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-06-22T18:07:18.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f088d485700 1 --2- 172.21.15.110:0/970673675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f088815a710 0x7f08880c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f088cc84700 1 --2- 172.21.15.110:0/970673675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088806a080 0x7f0888169870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f087ffff700 1 -- 172.21.15.110:0/970673675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f088815a710 msgr2=0x7f08880c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f087ffff700 1 --2- 172.21.15.110:0/970673675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f088815a710 0x7f08880c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f087ffff700 1 -- 172.21.15.110:0/970673675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088806a080 msgr2=0x7f0888169870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-22T18:07:18.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f087ffff700 1 --2- 172.21.15.110:0/970673675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088806a080 0x7f0888169870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-22T18:07:18.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f087ffff700 1 -- 172.21.15.110:0/970673675 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08780094f0 con 0x7f08880cee50 2023-06-22T18:07:18.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.782+0000 7f088d485700 1 --2- 172.21.15.110:0/970673675 >> [v2:172.21.15.116:3301/0,v1:172.21.15.116:6790/0] conn(0x7f088815a710 0x7f08880c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-22T18:07:18.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.783+0000 7f087ffff700 1 --2- 172.21.15.110:0/970673675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08880cee50 0x7f088815e120 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0888089250 tx=0x7f0878017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.783+0000 7f087cff9700 1 -- 172.21.15.110:0/970673675 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878024440 con 0x7f08880cee50 2023-06-22T18:07:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.783+0000 7f087cff9700 1 -- 172.21.15.110:0/970673675 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08780245c0 con 0x7f08880cee50 2023-06-22T18:07:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.783+0000 7f088dc86700 1 -- 172.21.15.110:0/970673675 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08880c7a40 con 0x7f08880cee50 2023-06-22T18:07:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.783+0000 7f087cff9700 1 -- 172.21.15.110:0/970673675 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878024980 con 0x7f08880cee50 2023-06-22T18:07:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.783+0000 7f088dc86700 1 -- 172.21.15.110:0/970673675 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f088815e660 con 0x7f08880cee50 2023-06-22T18:07:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.784+0000 7f088dc86700 1 -- 172.21.15.110:0/970673675 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088815e850 con 0x7f08880cee50 2023-06-22T18:07:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.784+0000 7f087cff9700 1 -- 172.21.15.110:0/970673675 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f087801a420 con 0x7f08880cee50 2023-06-22T18:07:18.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.784+0000 7f087cff9700 1 --2- 172.21.15.110:0/970673675 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f0874066e90 0x7f0874069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-22T18:07:18.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.785+0000 7f088cc84700 1 --2- 172.21.15.110:0/970673675 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f0874066e90 0x7f0874069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-22T18:07:18.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.785+0000 7f087cff9700 1 -- 172.21.15.110:0/970673675 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f0878086430 con 0x7f08880cee50 2023-06-22T18:07:18.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.785+0000 7f088cc84700 1 --2- 172.21.15.110:0/970673675 >> [v2:172.21.15.110:6832/117671,v1:172.21.15.110:6833/117671] conn(0x7f0874066e90 0x7f0874069340 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f08840075a0 tx=0x7f08840097a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-22T18:07:18.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-06-22T18:07:18.789+0000 7f087cff9700 1 -- 172.21.15.110:0/970673675 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f0878053600 con 0x7f08880cee50 2023-06-22T18:07:19.021 INFO:teuthology.orchestra.run.smithi110.stderr:no valid command found; 1 closest matches: 2023-06-22T18:07:19.021 INFO:teuthology.orchestra.run.smithi110.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-22T18:07:19.021 INFO:teuthology.orchestra.run.smithi110.stderr:Error EINVAL: invalid command 2023-06-22T18:07:19.024 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-22T18:07:19.025 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-22T18:07:19.025 INFO:teuthology.misc:Shutting down mds daemons... 2023-06-22T18:07:19.026 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-22T18:07:19.026 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.034 INFO:tasks.ceph.mds.a:Stopped 2023-06-22T18:07:19.035 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-22T18:07:19.036 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.082 INFO:tasks.ceph.mds.c:Stopped 2023-06-22T18:07:19.083 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-22T18:07:19.083 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.094 INFO:tasks.ceph.mds.b:Stopped 2023-06-22T18:07:19.095 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-22T18:07:19.095 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.146 INFO:tasks.ceph.mds.d:Stopped 2023-06-22T18:07:19.147 INFO:teuthology.misc:Shutting down osd daemons... 2023-06-22T18:07:19.147 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-06-22T18:07:19.147 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.185 INFO:tasks.ceph.osd.0:Stopped 2023-06-22T18:07:19.186 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-06-22T18:07:19.186 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.251 INFO:tasks.ceph.osd.1:Stopped 2023-06-22T18:07:19.251 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-06-22T18:07:19.252 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.320 INFO:tasks.ceph.osd.2:Stopped 2023-06-22T18:07:19.320 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-06-22T18:07:19.321 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.383 INFO:tasks.ceph.osd.3:Stopped 2023-06-22T18:07:19.384 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-06-22T18:07:19.384 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.421 INFO:tasks.ceph.osd.4:Stopped 2023-06-22T18:07:19.422 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-06-22T18:07:19.422 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.490 INFO:tasks.ceph.osd.5:Stopped 2023-06-22T18:07:19.491 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-06-22T18:07:19.491 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.560 INFO:tasks.ceph.osd.6:Stopped 2023-06-22T18:07:19.561 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-06-22T18:07:19.561 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.629 INFO:tasks.ceph.osd.7:Stopped 2023-06-22T18:07:19.630 INFO:teuthology.misc:Shutting down mgr daemons... 2023-06-22T18:07:19.630 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-06-22T18:07:19.631 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.654 INFO:tasks.ceph.mgr.y:Stopped 2023-06-22T18:07:19.655 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-06-22T18:07:19.655 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.694 INFO:tasks.ceph.mgr.x:Stopped 2023-06-22T18:07:19.694 INFO:teuthology.misc:Shutting down mon daemons... 2023-06-22T18:07:19.695 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-06-22T18:07:19.695 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.704 INFO:tasks.ceph.mon.a:Stopped 2023-06-22T18:07:19.705 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-06-22T18:07:19.705 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.744 INFO:tasks.ceph.mon.b:Stopped 2023-06-22T18:07:19.745 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-06-22T18:07:19.745 INFO:teuthology.orchestra.run:waiting for 300 2023-06-22T18:07:19.795 INFO:tasks.ceph.mon.c:Stopped 2023-06-22T18:07:19.795 INFO:tasks.ceph:Checking cluster log for badness... 2023-06-22T18:07:19.796 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2023-06-22T18:07:19.825 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi110.front.sepia.ceph.com 2023-06-22T18:07:19.825 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-06-22T18:07:20.206 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi110.front.sepia.ceph.com 2023-06-22T18:07:20.206 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-06-22T18:07:20.297 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi110.front.sepia.ceph.com 2023-06-22T18:07:20.298 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-06-22T18:07:20.356 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi110.front.sepia.ceph.com 2023-06-22T18:07:20.356 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-06-22T18:07:20.409 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi116.front.sepia.ceph.com 2023-06-22T18:07:20.410 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-06-22T18:07:20.761 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi116.front.sepia.ceph.com 2023-06-22T18:07:20.761 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-06-22T18:07:20.932 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi116.front.sepia.ceph.com 2023-06-22T18:07:20.933 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-06-22T18:07:20.999 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi116.front.sepia.ceph.com 2023-06-22T18:07:21.000 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-06-22T18:07:21.086 INFO:tasks.ceph:Archiving mon data... 2023-06-22T18:07:21.088 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857/data/mon.a.tgz 2023-06-22T18:07:21.088 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-06-22T18:07:21.103 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.BNS8Sp3W1f 2023-06-22T18:07:21.103 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.BNS8Sp3W1f 2023-06-22T18:07:21.219 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.BNS8Sp3W1f 2023-06-22T18:07:21.299 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.BNS8Sp3W1f is 196KB 2023-06-22T18:07:21.320 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.BNS8Sp3W1f 2023-06-22T18:07:21.335 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857/data/mon.b.tgz 2023-06-22T18:07:21.335 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-06-22T18:07:21.351 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.oWTleVa0lv 2023-06-22T18:07:21.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.oWTleVa0lv 2023-06-22T18:07:21.476 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.oWTleVa0lv 2023-06-22T18:07:21.556 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.oWTleVa0lv is 274KB 2023-06-22T18:07:21.648 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.oWTleVa0lv 2023-06-22T18:07:21.662 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857/data/mon.c.tgz 2023-06-22T18:07:21.663 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2023-06-22T18:07:21.715 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.2zCvKGo5v7 2023-06-22T18:07:21.716 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.2zCvKGo5v7 2023-06-22T18:07:21.823 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.2zCvKGo5v7 2023-06-22T18:07:21.903 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.2zCvKGo5v7 is 274KB 2023-06-22T18:07:21.996 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.2zCvKGo5v7 2023-06-22T18:07:22.010 INFO:tasks.ceph:Cleaning ceph cluster... 2023-06-22T18:07:22.011 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-22T18:07:22.013 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-22T18:07:22.073 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-06-22T18:07:22.074 DEBUG:teuthology.orchestra.run.smithi110:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-22T18:07:22.078 DEBUG:teuthology.orchestra.run.smithi116:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-22T18:07:22.101 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-22T18:07:22.138 INFO:teuthology.orchestra.run.smithi116.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-22T18:07:22.144 INFO:tasks.ceph:Archiving crash dumps... 2023-06-22T18:07:22.145 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857/remote/smithi110/crash 2023-06-22T18:07:22.146 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-22T18:07:22.173 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857/remote/smithi116/crash 2023-06-22T18:07:22.178 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-22T18:07:22.207 INFO:tasks.ceph:Shutting down logrotate 2023-06-22T18:07:22.207 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T18:07:22.222 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-22T18:07:22.325 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-22T18:07:22.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-22T18:07:22.373 INFO:tasks.ceph:Compressing logs... 2023-06-22T18:07:22.374 DEBUG:teuthology.orchestra.run.smithi110:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T18:07:22.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T18:07:22.664 INFO:tasks.ceph:Archiving logs... 2023-06-22T18:07:22.665 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857/remote/smithi110/log 2023-06-22T18:07:22.666 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-22T18:07:22.729 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857/remote/smithi116/log 2023-06-22T18:07:22.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-22T18:07:22.792 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-22T18:07:22.868 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c33b3a9a87444a4e8029701c4f04a7a8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-22T18:07:22.874 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-06-22T18:07:22.883 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-06-22T18:07:22.928 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-06-22T18:07:22.929 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-22T18:07:22.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-22T18:07:23.035 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-06-22T18:07:23.043 INFO:teuthology.task.clock:Checking final clock skew... 2023-06-22T18:07:23.043 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T18:07:23.046 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-22T18:07:23.057 INFO:teuthology.orchestra.run.smithi110.stderr:bash: ntpq: command not found 2023-06-22T18:07:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:bash: ntpq: command not found 2023-06-22T18:07:23.059 INFO:teuthology.orchestra.run.smithi110.stdout:210 Number of sources = 4 2023-06-22T18:07:23.060 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T18:07:23.060 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2023-06-22T18:07:23.060 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv01.front.sepia.ceph.com 2 6 177 46 +1336us[+1336us] +/- 106ms 2023-06-22T18:07:23.060 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv02.front.sepia.ceph.com 3 6 177 46 -3113us[-3121us] +/- 79ms 2023-06-22T18:07:23.061 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 177 47 +2347us[+2339us] +/- 88ms 2023-06-22T18:07:23.061 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T18:07:23.062 INFO:teuthology.orchestra.run.smithi116.stdout:210 Number of sources = 4 2023-06-22T18:07:23.062 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-22T18:07:23.063 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2023-06-22T18:07:23.063 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 6 177 45 +1331us[+1331us] +/- 106ms 2023-06-22T18:07:23.063 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 177 45 -3116us[-3105us] +/- 79ms 2023-06-22T18:07:23.063 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 177 47 +2356us[+2367us] +/- 88ms 2023-06-22T18:07:23.063 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-22T18:07:23.064 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-06-22T18:07:23.077 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-06-22T18:07:23.078 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-06-22T18:07:23.124 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-22T18:07:23.209 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-22T18:07:23.295 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T18:07:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T18:07:23.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-22T18:07:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T18:07:23.355 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-06-22T18:07:23.365 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-06-22T18:07:23.427 INFO:teuthology.task.internal:Duration was 976.111125 seconds 2023-06-22T18:07:23.428 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-06-22T18:07:23.461 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-06-22T18:07:23.462 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-22T18:07:23.464 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-22T18:07:23.505 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T18:07:23.508 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-22T18:07:23.927 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-06-22T18:07:23.927 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2023-06-22T18:07:23.928 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-22T18:07:23.946 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2023-06-22T18:07:23.947 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-22T18:07:23.965 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-06-22T18:07:23.966 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T18:07:23.988 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-22T18:07:24.043 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-06-22T18:07:24.044 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-22T18:07:24.065 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-22T18:07:24.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-06-22T18:07:24.170 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-06-22T18:07:24.170 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-22T18:07:24.194 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-22T18:07:24.219 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-06-22T18:07:24.230 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-22T18:07:24.237 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-22T18:07:24.258 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2023-06-22T18:07:24.283 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2023-06-22T18:07:24.296 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-22T18:07:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T18:07:24.326 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-22T18:07:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-22T18:07:24.351 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-06-22T18:07:24.362 INFO:teuthology.task.internal:Transferring archived files... 2023-06-22T18:07:24.363 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857/remote/smithi110 2023-06-22T18:07:24.363 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-22T18:07:24.400 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857/remote/smithi116 2023-06-22T18:07:24.401 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-22T18:07:24.433 INFO:teuthology.task.internal:Removing archive directory... 2023-06-22T18:07:24.434 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-22T18:07:24.436 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-22T18:07:24.482 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-06-22T18:07:24.494 INFO:teuthology.task.internal:Not uploading archives. 2023-06-22T18:07:24.494 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-06-22T18:07:24.538 INFO:teuthology.task.internal:Tidying up after the test... 2023-06-22T18:07:24.538 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-22T18:07:24.541 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-22T18:07:24.552 INFO:teuthology.orchestra.run.smithi110.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 22 18:07 /home/ubuntu/cephtest 2023-06-22T18:07:24.554 INFO:teuthology.orchestra.run.smithi116.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 22 18:07 /home/ubuntu/cephtest 2023-06-22T18:07:24.555 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-06-22T18:07:24.570 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-06-22T18:07:24.656 INFO:teuthology.nuke:Checking targets against current locks 2023-06-22T18:07:24.686 DEBUG:teuthology.nuke:shortname: smithi110 2023-06-22T18:07:24.687 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T18:07:24.711 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 17:43:43.671000', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE8qRhuRUXj1iFiQd/Gr7ahxAb/xrAR5aYNjNZNzG8lND1ManSzV5hIdYStVClXN02MXcaxubtpjWEjQm1azrps='} 2023-06-22T18:07:24.714 DEBUG:teuthology.nuke:shortname: smithi116 2023-06-22T18:07:24.715 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T18:07:24.738 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 17:43:43.672957', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOlY0LD4+AaVWVxqWRbI1ZOcEjeJnJyA/uEqjVQqVqKkgaSUHAk8XiDdozZLzJuZTuX+5Sc911PDBBIpIciEguKTBN1jMRTf4XX0SDPQbjZfukPSSEBZGxWzuVyfPk1UlCGdlHbkGHH0xwvWBerXFjA7qSH4rXOR7NizG70cnbCdih0S/hzVNGJDLVSvJ5gAiqE6PxjQ7Zf3zLNiwzvMhD4dolr7HwhSnAOddGaqSo8ZU+6VM+BYWFy2syLN7tDV1f8UCYt26ItcV6MMvwQRI4usWpNssQnI8J2j2Dn2PmPBdNFnmwFp4UyKK7Aq0lT032ygSvVdD7Oa5ZKlFg1UaZcNXfAZudG/qXwN0rzebd4cvKVq0Hw3jpHiTLqYGqQtE2Cz1iFmmoD7LSoVBXGItMBZJw+L+UhlkM0+wXAk/44WSV0CRMOcR8aDx5NiCnHv5Vw3NED+SpkjGsLzFTwwWQQoWIf39TilTSGqpZrWnp9ulvrgIv7VAVEAD6fiaVPqU='} 2023-06-22T18:07:24.767 INFO:teuthology.orchestra.console:Power off smithi110 2023-06-22T18:07:24.768 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-22T18:07:24.791 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T18:07:24.816 INFO:teuthology.orchestra.console:Power off smithi116 2023-06-22T18:07:24.817 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-22T18:07:24.842 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T18:07:28.830 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T18:07:28.869 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T18:07:32.944 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T18:07:32.981 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T18:07:33.158 INFO:teuthology.orchestra.console:Power off for smithi110 completed 2023-06-22T18:07:37.098 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T18:07:37.315 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2023-06-22T18:07:37.420 INFO:teuthology.run:Summary data: description: fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} duration: 976.111124753952 failure_reason: 'Command failed on smithi110 with status 22: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c33b3a9a87444a4e8029701c4f04a7a8 status: fail success: false 2023-06-22T18:07:37.421 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T18:07:37.509 INFO:teuthology.run:FAIL