2023-06-23T00:15:50.922 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-23T00:15:50.923 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876/orig.config.yaml 2023-06-23T00:15:50.968 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T00:15:51.056 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876 branch: wip-yuri4-testing-2023-06-21-0846-quincy description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} email: yweinste@redhat.com first_in_suite: false job_id: '7311876' kernel: client: sha1: distro kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-06-21-0846-quincy ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - bad backtrace - object missing on disk - error reading table object - error reading sessionmap - unmatched fragstat - unmatched rstat - was unreadable, recreating it now - Scrub error on inode - Metadata damage detected - MDS_FAILED - MDS_DAMAGE sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-06-21-0846-quincy sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 9037 sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2023-06-21-0846-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI35AffQKiJerBXQiexJsHNi/eRFM1jEN4uDgZt8gEo24uO9C7Qevh+1aUsEOcUiyFnWYA8rSQfZ+ukerdf7UXM= smithi086.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5jJYX/k8e04S5zh43QvZfHdNrNLdUA9OK/jgcrj2wjYuUsSN+K3Oo5ocRKvqb4CqxrGPGLjG2JmlD44PlF3+NxFjcFM/JdCe/A3Cj+/w2m09YrQUPEio0hSfLNCYGdAGmVdD/1U3urI8Z0eWooe3gQCj6E3+Qkh8/gVa/JQUYIrnsvI7OU/ZMSXrbhmBKy164mYKMTQBClxOV4Yb9kPf4CeNWK5SF5KrYu7n3nGO2CThcRomrpdBl/W8h0ZEbD3ALzmANcLgfJCZZuKdUM7xtCelXLEtvEqtinRTSelX6Xo7f859ZOUcIn3V1jQ6dpG6GxnF58IrXQI4llkJWfo00MvtlMZ7ZMgh3RRA97j6M+I8LrHayFNDE2Rk6eCOgfnQ5BLvu93X7VVmSBmUBWCwEpHOaMSkPnH+lTZkvTdjnyn8capN1C8Rpnr6H/27RTccw20khijHhhdvrUh2oisPBw4vGLD1WdfW2bpTIQXQggaV3S/Cvy+rVKBPzp2ZIkTE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_recovery_pool teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 076bbebc42a14f7d568aaa78eabb0038327bcb23 timestamp: 2023-06-22_15:25:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-06-23T00:15:51.056 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa; will attempt to use it 2023-06-23T00:15:51.057 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks 2023-06-23T00:15:51.057 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-06-23T00:15:51.059 INFO:teuthology.task.internal:Checking packages... 2023-06-23T00:15:51.087 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '42ddc46e6e6582d8e29a5a982a81886fe028be2c' 2023-06-23T00:15:51.087 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-06-23T00:15:51.088 INFO:teuthology.packaging:ref: None 2023-06-23T00:15:51.088 INFO:teuthology.packaging:tag: None 2023-06-23T00:15:51.088 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T00:15:51.088 INFO:teuthology.packaging:sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:15:51.089 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T00:15:51.311 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-719.g42ddc46e 2023-06-23T00:15:51.311 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-06-23T00:15:51.320 INFO:teuthology.task.internal:no buildpackages task found 2023-06-23T00:15:51.320 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-06-23T00:15:51.331 INFO:teuthology.task.internal:Saving configuration 2023-06-23T00:15:51.347 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-06-23T00:15:51.353 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T00:15:51.380 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:10:53.609891', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI35AffQKiJerBXQiexJsHNi/eRFM1jEN4uDgZt8gEo24uO9C7Qevh+1aUsEOcUiyFnWYA8rSQfZ+ukerdf7UXM='} 2023-06-23T00:15:51.404 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:10:53.611194', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFN9qO3J1U3m0yE6VWK3gQpm0jZ7aOU17FrkdKcAgDqPEUvdGHuoWcnuMSI3cryzFJphSlgUNjvyrFOEZfum8l8='} 2023-06-23T00:15:51.405 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-06-23T00:15:51.410 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-06-23T00:15:51.411 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-06-23T00:15:51.411 INFO:teuthology.run_tasks:Running task console_log... 2023-06-23T00:15:51.485 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff78dc84040>, signals=[15]) 2023-06-23T00:15:51.486 INFO:teuthology.run_tasks:Running task internal.connect... 2023-06-23T00:15:51.493 INFO:teuthology.task.internal:Opening connections... 2023-06-23T00:15:51.494 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:15:51.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:15:51.563 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2023-06-23T00:15:51.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:15:51.632 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-06-23T00:15:51.644 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2023-06-23T00:15:51.659 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2023-06-23T00:15:51.660 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2023-06-23T00:15:51.713 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="Red Hat Enterprise Linux" 2023-06-23T00:15:51.714 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="8.4 (Ootpa)" 2023-06-23T00:15:51.714 INFO:teuthology.orchestra.run.smithi064.stdout:ID="rhel" 2023-06-23T00:15:51.714 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="fedora" 2023-06-23T00:15:51.714 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="8.4" 2023-06-23T00:15:51.714 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el8" 2023-06-23T00:15:51.714 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-23T00:15:51.714 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2023-06-23T00:15:51.715 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-23T00:15:51.715 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://www.redhat.com/" 2023-06-23T00:15:51.715 INFO:teuthology.orchestra.run.smithi064.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-23T00:15:51.715 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T00:15:51.715 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:15:51.715 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T00:15:51.715 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-23T00:15:51.715 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-23T00:15:51.716 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-23T00:15:51.716 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2023-06-23T00:15:51.739 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2023-06-23T00:15:51.758 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2023-06-23T00:15:51.758 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2023-06-23T00:15:51.814 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="Red Hat Enterprise Linux" 2023-06-23T00:15:51.815 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8.4 (Ootpa)" 2023-06-23T00:15:51.815 INFO:teuthology.orchestra.run.smithi086.stdout:ID="rhel" 2023-06-23T00:15:51.816 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="fedora" 2023-06-23T00:15:51.816 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8.4" 2023-06-23T00:15:51.816 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8" 2023-06-23T00:15:51.816 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-23T00:15:51.816 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2023-06-23T00:15:51.817 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-23T00:15:51.817 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://www.redhat.com/" 2023-06-23T00:15:51.817 INFO:teuthology.orchestra.run.smithi086.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-23T00:15:51.817 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T00:15:51.817 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:15:51.817 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T00:15:51.817 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-23T00:15:51.818 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-23T00:15:51.818 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-23T00:15:51.818 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2023-06-23T00:15:51.839 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-06-23T00:15:51.854 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-06-23T00:15:51.860 INFO:teuthology.task.internal:Checking for old test directory... 2023-06-23T00:15:51.861 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2023-06-23T00:15:51.863 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2023-06-23T00:15:51.879 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-06-23T00:15:51.887 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-06-23T00:15:51.887 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2023-06-23T00:15:51.918 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2023-06-23T00:15:51.930 INFO:teuthology.orchestra.run.smithi064.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-23T00:15:51.935 INFO:teuthology.orchestra.run.smithi086.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-23T00:15:51.935 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-06-23T00:15:51.991 INFO:teuthology.run_tasks:Running task kernel... 2023-06-23T00:15:52.008 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-06-23T00:15:52.009 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-06-23T00:15:52.010 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-06-23T00:15:52.010 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T00:15:52.011 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T00:15:52.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:15:52.024 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2023-06-23T00:15:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:15:52.026 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2023-06-23T00:15:52.079 INFO:teuthology.orchestra.run.smithi064.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:15:52.079 INFO:teuthology.task.kernel:Running kernel on smithi064: 4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:15:52.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2023-06-23T00:15:52.081 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:15:52.082 INFO:teuthology.task.kernel:Running kernel on smithi086: 4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:15:52.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2023-06-23T00:15:53.482 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-06-23T00:15:53.483 INFO:teuthology.orchestra.run.smithi086.stdout:Unable to read consumer identity 2023-06-23T00:15:53.577 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-06-23T00:15:53.577 INFO:teuthology.orchestra.run.smithi064.stdout:Unable to read consumer identity 2023-06-23T00:15:57.550 INFO:teuthology.orchestra.run.smithi064.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T00:15:57.603 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 1.2 kB/s | 3.5 kB 00:02 2023-06-23T00:18:11.361 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 120 kB/s | 16 MB 02:13 2023-06-23T00:18:12.245 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 119 kB/s | 16 MB 02:14 2023-06-23T00:18:14.587 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 3.4 MB/s | 24 kB 00:00 2023-06-23T00:18:15.315 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-23T00:18:15.315 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-23T00:18:15.315 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-23T00:18:15.458 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-06-23T00:18:15.460 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2023-06-23T00:18:15.460 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-06-23T00:18:15.563 DEBUG:teuthology.orchestra.run.smithi086:> echo no | sudo yum reinstall kernel || true 2023-06-23T00:18:15.642 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2023-06-23T00:18:15.908 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-06-23T00:18:15.909 INFO:teuthology.orchestra.run.smithi086.stdout:Unable to read consumer identity 2023-06-23T00:18:16.197 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 12:18:14 AM UTC. 2023-06-23T00:18:16.331 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:18:16.331 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:18:16.331 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:18:16.354 INFO:teuthology.orchestra.run.smithi086.stderr:Error: No packages marked for reinstall. 2023-06-23T00:18:16.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum reinstall -y kernel || true 2023-06-23T00:18:16.410 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-23T00:18:16.411 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-23T00:18:16.411 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-23T00:18:16.558 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2023-06-23T00:18:16.559 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2023-06-23T00:18:16.560 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2023-06-23T00:18:16.655 DEBUG:teuthology.orchestra.run.smithi064:> echo no | sudo yum reinstall kernel || true 2023-06-23T00:18:16.746 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-06-23T00:18:16.747 INFO:teuthology.orchestra.run.smithi086.stdout:Unable to read consumer identity 2023-06-23T00:18:17.023 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-06-23T00:18:17.024 INFO:teuthology.orchestra.run.smithi064.stdout:Unable to read consumer identity 2023-06-23T00:18:17.029 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:03 ago on Fri 23 Jun 2023 12:18:14 AM UTC. 2023-06-23T00:18:17.163 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:18:17.163 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:18:17.163 INFO:teuthology.orchestra.run.smithi086.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:18:17.176 INFO:teuthology.orchestra.run.smithi086.stderr:Error: No packages marked for reinstall. 2023-06-23T00:18:17.224 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:18:17.332 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 12:18:15 AM UTC. 2023-06-23T00:18:17.349 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:17.350 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:17.350 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:17.350 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T00:18:17.351 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T00:18:17.351 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2023-06-23T00:18:17.351 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T00:18:17.352 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:18:17.467 INFO:teuthology.orchestra.run.smithi064.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:18:17.467 INFO:teuthology.orchestra.run.smithi064.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:18:17.468 INFO:teuthology.orchestra.run.smithi064.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:18:17.476 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:17.477 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:17.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2023-06-23T00:18:17.496 INFO:teuthology.orchestra.run.smithi064.stderr:Error: No packages marked for reinstall. 2023-06-23T00:18:17.504 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2023-06-23T00:18:17.504 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2023-06-23T00:18:17.504 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.02 2023-06-23T00:18:17.505 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 99.el8_4.1 2023-06-23T00:18:17.505 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2023-06-23T00:18:17.505 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-23T00:18:17.505 INFO:teuthology.orchestra.run.smithi086.stdout:Group : System Environment/Base 2023-06-23T00:18:17.505 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 9264631 2023-06-23T00:18:17.505 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2023-06-23T00:18:17.505 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-23T00:18:17.506 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-23T00:18:17.506 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-23T00:18:17.506 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-23T00:18:17.506 INFO:teuthology.orchestra.run.smithi086.stdout:Relocations : (not relocatable) 2023-06-23T00:18:17.506 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : Red Hat, Inc. 2023-06-23T00:18:17.506 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : Red Hat, Inc. 2023-06-23T00:18:17.507 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T00:18:17.507 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2023-06-23T00:18:17.507 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2023-06-23T00:18:17.507 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:18:17.507 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T00:18:17.507 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T00:18:17.507 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T00:18:17.507 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2023-06-23T00:18:17.508 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:18:17.508 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T00:18:17.509 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T00:18:17.509 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:17.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T00:18:17.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum reinstall -y kernel || true 2023-06-23T00:18:17.912 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-06-23T00:18:17.912 INFO:teuthology.orchestra.run.smithi064.stdout:Unable to read consumer identity 2023-06-23T00:18:18.207 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:03 ago on Fri 23 Jun 2023 12:18:15 AM UTC. 2023-06-23T00:18:18.341 INFO:teuthology.orchestra.run.smithi064.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:18:18.341 INFO:teuthology.orchestra.run.smithi064.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:18:18.342 INFO:teuthology.orchestra.run.smithi064.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:18:18.351 INFO:teuthology.orchestra.run.smithi064.stderr:Error: No packages marked for reinstall. 2023-06-23T00:18:18.406 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:18:18.522 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2023-06-23T00:18:18.537 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:18.537 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:18.538 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:18.538 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T00:18:18.538 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T00:18:18.538 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-06-23T00:18:18.538 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T00:18:18.538 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:18:18.668 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:18.668 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:18.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2023-06-23T00:18:18.697 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2023-06-23T00:18:18.697 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2023-06-23T00:18:18.697 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.02 2023-06-23T00:18:18.697 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 99.el8_4.1 2023-06-23T00:18:18.698 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2023-06-23T00:18:18.698 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-23T00:18:18.698 INFO:teuthology.orchestra.run.smithi064.stdout:Group : System Environment/Base 2023-06-23T00:18:18.698 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 9264631 2023-06-23T00:18:18.698 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2023-06-23T00:18:18.698 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-23T00:18:18.698 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-23T00:18:18.699 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-23T00:18:18.699 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-23T00:18:18.699 INFO:teuthology.orchestra.run.smithi064.stdout:Relocations : (not relocatable) 2023-06-23T00:18:18.699 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : Red Hat, Inc. 2023-06-23T00:18:18.699 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : Red Hat, Inc. 2023-06-23T00:18:18.699 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T00:18:18.699 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2023-06-23T00:18:18.699 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2023-06-23T00:18:18.700 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:18:18.700 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T00:18:18.700 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T00:18:18.700 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T00:18:18.700 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2023-06-23T00:18:18.700 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:18:18.701 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T00:18:18.701 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T00:18:18.702 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:18.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T00:18:19.733 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2023-06-23T00:18:20.475 INFO:teuthology.orchestra.run.smithi086.stderr:done 2023-06-23T00:18:20.476 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2023-06-23T00:18:20.491 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.0I4nLy9em2 2023-06-23T00:18:20.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0I4nLy9em2 2023-06-23T00:18:20.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.0I4nLy9em2 2023-06-23T00:18:20.736 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.0I4nLy9em2 is 6KB 2023-06-23T00:18:20.785 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.0I4nLy9em2 2023-06-23T00:18:20.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T00:18:20.873 INFO:teuthology.orchestra.run.smithi086.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-23T00:18:20.873 INFO:teuthology.orchestra.run.smithi086.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-23T00:18:20.873 INFO:teuthology.orchestra.run.smithi086.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-23T00:18:20.873 INFO:teuthology.orchestra.run.smithi086.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-23T00:18:20.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:21.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2023-06-23T00:18:21.437 INFO:teuthology.orchestra.run.smithi064.stderr:done 2023-06-23T00:18:21.439 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2023-06-23T00:18:21.455 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.Y5MUnNDR3y 2023-06-23T00:18:21.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Y5MUnNDR3y 2023-06-23T00:18:21.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.Y5MUnNDR3y 2023-06-23T00:18:21.675 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.Y5MUnNDR3y is 6KB 2023-06-23T00:18:21.722 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.Y5MUnNDR3y 2023-06-23T00:18:21.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T00:18:21.803 INFO:teuthology.orchestra.run.smithi064.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-23T00:18:21.804 INFO:teuthology.orchestra.run.smithi064.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-23T00:18:21.804 INFO:teuthology.orchestra.run.smithi064.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-23T00:18:21.804 INFO:teuthology.orchestra.run.smithi064.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-23T00:18:21.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:18:21.956 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2023-06-23T00:18:51.068 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-06-23T00:18:51.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-06-23T00:18:51.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:18:51.960 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-06-23T00:18:51.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-06-23T00:18:51.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:19:09.552 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2023-06-23T00:19:10.320 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2023-06-23T00:19:39.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-06-23T00:19:39.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:19:40.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-06-23T00:19:40.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:19:57.936 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2023-06-23T00:19:58.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2023-06-23T00:20:27.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-06-23T00:20:27.965 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:20:28.363 DEBUG:teuthology.orchestra.run.smithi086:> true 2023-06-23T00:20:28.646 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-06-23T00:20:28.647 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-23T00:20:28.647 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2023-06-23T00:20:28.662 INFO:teuthology.orchestra.run.smithi086.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-23T00:20:28.663 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:20:28.663 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T00:20:28.663 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-23T00:20:28.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-06-23T00:20:28.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:20:29.070 DEBUG:teuthology.orchestra.run.smithi064:> true 2023-06-23T00:20:29.349 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-06-23T00:20:29.349 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-23T00:20:29.349 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2023-06-23T00:20:29.368 INFO:teuthology.orchestra.run.smithi064.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-23T00:20:29.368 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:20:29.368 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T00:20:29.369 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-23T00:20:29.665 DEBUG:teuthology.parallel:result is None 2023-06-23T00:20:30.369 DEBUG:teuthology.parallel:result is None 2023-06-23T00:20:30.370 INFO:teuthology.run_tasks:Running task internal.base... 2023-06-23T00:20:30.378 INFO:teuthology.task.internal:Creating test directory... 2023-06-23T00:20:30.379 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T00:20:30.381 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T00:20:30.397 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-06-23T00:20:30.435 INFO:teuthology.run_tasks:Running task internal.archive... 2023-06-23T00:20:30.448 INFO:teuthology.task.internal:Creating archive directory... 2023-06-23T00:20:30.448 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T00:20:30.451 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T00:20:30.487 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-06-23T00:20:30.499 INFO:teuthology.task.internal:Enabling coredump saving... 2023-06-23T00:20:30.500 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T00:20:30.519 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T00:20:30.613 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:20:30.629 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:20:30.638 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:20:30.649 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:20:30.650 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-06-23T00:20:30.659 INFO:teuthology.task.internal:Configuring sudo... 2023-06-23T00:20:30.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T00:20:30.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T00:20:30.689 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-06-23T00:20:30.701 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-06-23T00:20:30.702 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T00:20:30.732 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T00:20:30.749 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:20:30.833 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:20:30.871 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:20:30.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:20:30.997 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:20:30.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T00:20:31.061 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:20:31.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:20:31.155 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:20:31.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:20:31.288 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:20:31.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T00:20:31.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2023-06-23T00:20:31.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2023-06-23T00:20:31.429 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T00:20:31.473 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T00:20:31.912 INFO:teuthology.run_tasks:Running task internal.timer... 2023-06-23T00:20:31.926 INFO:teuthology.task.internal:Starting timer... 2023-06-23T00:20:31.926 INFO:teuthology.run_tasks:Running task pcp... 2023-06-23T00:20:31.951 INFO:teuthology.run_tasks:Running task selinux... 2023-06-23T00:20:31.966 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-06-23T00:20:31.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2023-06-23T00:20:32.040 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: [ OK ] 2023-06-23T00:20:32.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2023-06-23T00:20:32.114 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: [ OK ] 2023-06-23T00:20:32.114 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-06-23T00:20:32.115 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2023-06-23T00:20:32.145 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2023-06-23T00:20:32.145 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2023-06-23T00:20:32.182 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2023-06-23T00:20:32.183 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2023-06-23T00:20:32.183 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T00:20:32.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:20:32.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T00:20:32.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:20:32.256 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-06-23T00:20:32.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2023-06-23T00:20:32.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2023-06-23T00:20:32.325 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-06-23T00:20:32.366 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-06-23T00:20:32.367 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-06-23T00:20:32.402 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-06-23T00:20:32.405 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-06-23T00:30:49.970 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2023-06-23T00:30:49.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-06-23T00:30:49.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:30:50.042 DEBUG:teuthology.orchestra.run.smithi064:> true 2023-06-23T00:30:50.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2023-06-23T00:30:50.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:30:50.177 DEBUG:teuthology.orchestra.run.smithi086:> true 2023-06-23T00:30:50.242 INFO:teuthology.run_tasks:Running task clock... 2023-06-23T00:30:50.265 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-06-23T00:30:50.266 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T00:30:50.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T00:30:50.271 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T00:30:50.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T00:30:50.299 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T00:30:50.311 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T00:30:50.314 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T00:30:50.328 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T00:30:50.354 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2023-06-23T00:30:50.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2023-06-23T00:30:50.364 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2023-06-23T00:30:50.368 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2023-06-23T00:30:50.379 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T00:30:50.382 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T00:30:50.393 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T00:30:50.395 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T00:30:50.455 INFO:teuthology.orchestra.run.smithi064.stderr:bash: ntpq: command not found 2023-06-23T00:30:50.458 INFO:teuthology.orchestra.run.smithi064.stdout:210 Number of sources = 4 2023-06-23T00:30:50.458 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T00:30:50.459 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2023-06-23T00:30:50.459 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:30:50.459 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:30:50.459 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:30:50.459 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:30:50.461 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2023-06-23T00:30:50.463 INFO:teuthology.orchestra.run.smithi086.stdout:210 Number of sources = 4 2023-06-23T00:30:50.463 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T00:30:50.463 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2023-06-23T00:30:50.464 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:30:50.464 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:30:50.464 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:30:50.464 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:30:50.465 INFO:teuthology.run_tasks:Running task install... 2023-06-23T00:30:50.474 DEBUG:teuthology.task.install:project ceph 2023-06-23T00:30:50.475 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}} 2023-06-23T00:30:50.475 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'} 2023-06-23T00:30:50.475 INFO:teuthology.task.install:Using flavor: default 2023-06-23T00:30:50.481 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-06-23T00:30:50.482 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-06-23T00:30:50.483 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T00:30:50.484 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:30:50.488 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T00:30:50.489 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:30:50.638 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T00:30:50.638 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T00:30:50.702 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T00:30:50.702 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T00:30:50.809 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T00:30:50.809 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:30:50.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T00:30:50.836 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T00:30:50.837 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T00:30:50.867 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T00:30:50.868 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:30:50.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T00:30:50.895 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T00:30:50.895 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T00:30:50.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T00:30:50.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T00:30:50.975 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T00:30:51.038 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2023-06-23T00:30:51.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2023-06-23T00:30:51.041 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T00:30:51.105 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2023-06-23T00:30:51.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2023-06-23T00:30:51.431 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-06-23T00:30:51.517 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-06-23T00:30:51.782 INFO:teuthology.orchestra.run.smithi086.stdout:44 files removed 2023-06-23T00:30:51.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T00:30:51.925 INFO:teuthology.orchestra.run.smithi064.stdout:44 files removed 2023-06-23T00:30:51.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T00:30:52.182 INFO:teuthology.orchestra.run.smithi086.stdout:Updating Subscription Management repositories. 2023-06-23T00:30:52.350 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-06-23T00:30:53.041 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 233 kB/s | 78 kB 00:00 2023-06-23T00:30:53.205 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 209 kB/s | 78 kB 00:00 2023-06-23T00:30:53.330 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 37 kB/s | 9.2 kB 00:00 2023-06-23T00:30:53.553 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 30 kB/s | 9.2 kB 00:00 2023-06-23T00:30:53.604 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 5.9 kB/s | 1.4 kB 00:00 2023-06-23T00:30:53.825 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 6.0 kB/s | 1.4 kB 00:00 2023-06-23T00:30:55.893 INFO:teuthology.orchestra.run.smithi086.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-06-23T00:30:56.038 INFO:teuthology.orchestra.run.smithi064.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2023-06-23T00:30:56.611 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 29 MB/s | 16 MB 00:00 2023-06-23T00:31:00.062 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-06-23T00:31:00.507 INFO:teuthology.orchestra.run.smithi064.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 106 MB/s | 38 MB 00:00 2023-06-23T00:31:06.684 INFO:teuthology.orchestra.run.smithi064.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 55 MB/s | 33 MB 00:00 2023-06-23T00:31:15.094 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T00:31:15.095 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T00:31:15.106 INFO:teuthology.orchestra.run.smithi064.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-23T00:31:15.107 INFO:teuthology.orchestra.run.smithi064.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-23T00:31:15.108 INFO:teuthology.orchestra.run.smithi064.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-23T00:31:15.109 INFO:teuthology.orchestra.run.smithi064.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T00:31:15.111 INFO:teuthology.orchestra.run.smithi064.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T00:31:15.113 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-23T00:31:15.114 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-23T00:31:15.624 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2023-06-23T00:31:15.635 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================================================== 2023-06-23T00:31:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2023-06-23T00:31:15.636 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================================================== 2023-06-23T00:31:15.636 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2023-06-23T00:31:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-23T00:31:15.636 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T00:31:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T00:31:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T00:31:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T00:31:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T00:31:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T00:31:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T00:31:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T00:31:15.637 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T00:31:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T00:31:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T00:31:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T00:31:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T00:31:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T00:31:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T00:31:15.638 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-23T00:31:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-23T00:31:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-23T00:31:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T00:31:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T00:31:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T00:31:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-23T00:31:15.639 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-23T00:31:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-23T00:31:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T00:31:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T00:31:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T00:31:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T00:31:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T00:31:15.640 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T00:31:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T00:31:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-23T00:31:15.641 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2023-06-23T00:31:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T00:31:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T00:31:15.641 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2023-06-23T00:31:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-23T00:31:15.641 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T00:31:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T00:31:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T00:31:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T00:31:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T00:31:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T00:31:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T00:31:15.642 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T00:31:15.643 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-23T00:31:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T00:31:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-23T00:31:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-23T00:31:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-23T00:31:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T00:31:15.644 INFO:teuthology.orchestra.run.smithi064.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-23T00:31:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-23T00:31:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T00:31:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-23T00:31:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-23T00:31:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T00:31:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-23T00:31:15.645 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T00:31:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-23T00:31:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-23T00:31:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T00:31:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-23T00:31:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-23T00:31:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-23T00:31:15.646 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-23T00:31:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-23T00:31:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-23T00:31:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-23T00:31:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-23T00:31:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T00:31:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-23T00:31:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T00:31:15.647 INFO:teuthology.orchestra.run.smithi064.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T00:31:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T00:31:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T00:31:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T00:31:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T00:31:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T00:31:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T00:31:15.648 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T00:31:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T00:31:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T00:31:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-23T00:31:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-23T00:31:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-23T00:31:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T00:31:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T00:31:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T00:31:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-23T00:31:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-23T00:31:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T00:31:15.650 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T00:31:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-23T00:31:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-23T00:31:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-23T00:31:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T00:31:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T00:31:15.651 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-23T00:31:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-23T00:31:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-23T00:31:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-23T00:31:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T00:31:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T00:31:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T00:31:15.652 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T00:31:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-23T00:31:15.653 INFO:teuthology.orchestra.run.smithi064.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T00:31:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T00:31:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T00:31:15.654 INFO:teuthology.orchestra.run.smithi064.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T00:31:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T00:31:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T00:31:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T00:31:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T00:31:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-23T00:31:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T00:31:15.655 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-23T00:31:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T00:31:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-23T00:31:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-23T00:31:15.656 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:31:15.656 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2023-06-23T00:31:15.656 INFO:teuthology.orchestra.run.smithi064.stdout:====================================================================================================================== 2023-06-23T00:31:15.656 INFO:teuthology.orchestra.run.smithi064.stdout:Install 118 Packages 2023-06-23T00:31:15.656 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2023-06-23T00:31:15.657 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:31:15.657 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 201 M 2023-06-23T00:31:15.657 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2023-06-23T00:31:15.898 INFO:teuthology.orchestra.run.smithi064.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 42 kB/s | 6.4 kB 00:00 2023-06-23T00:31:16.300 INFO:teuthology.orchestra.run.smithi064.stdout:(2/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 9.3 MB/s | 5.2 MB 00:00 2023-06-23T00:31:16.342 INFO:teuthology.orchestra.run.smithi064.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 3.7 MB/s | 162 kB 00:00 2023-06-23T00:31:16.475 INFO:teuthology.orchestra.run.smithi064.stdout:(4/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 1.5 MB/s | 873 kB 00:00 2023-06-23T00:31:16.548 INFO:teuthology.orchestra.run.smithi064.stdout:(5/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 11 MB/s | 2.3 MB 00:00 2023-06-23T00:31:16.889 INFO:teuthology.orchestra.run.smithi064.stdout:(6/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 12 MB/s | 3.9 MB 00:00 2023-06-23T00:31:17.198 INFO:teuthology.orchestra.run.smithi064.stdout:(7/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 2.2 MB/s | 1.6 MB 00:00 2023-06-23T00:31:17.546 INFO:teuthology.orchestra.run.smithi064.stdout:(8/120): ceph-common-17.2.6-719.g42ddc46e.el8.x 13 MB/s | 23 MB 00:01 2023-06-23T00:31:17.585 INFO:teuthology.orchestra.run.smithi064.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 625 kB/s | 24 kB 00:00 2023-06-23T00:31:18.441 INFO:teuthology.orchestra.run.smithi064.stdout:(10/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 11 MB/s | 17 MB 00:01 2023-06-23T00:31:18.502 INFO:teuthology.orchestra.run.smithi064.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 4.1 MB/s | 255 kB 00:00 2023-06-23T00:31:18.819 INFO:teuthology.orchestra.run.smithi064.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 78 kB/s | 25 kB 00:00 2023-06-23T00:31:19.546 INFO:teuthology.orchestra.run.smithi064.stdout:(13/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 1.1 MB/s | 794 kB 00:00 2023-06-23T00:31:19.984 INFO:teuthology.orchestra.run.smithi064.stdout:(14/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 6.0 MB/s | 17 MB 00:02 2023-06-23T00:31:19.985 INFO:teuthology.orchestra.run.smithi064.stdout:(15/120): libcephsqlite-17.2.6-719.g42ddc46e.el 397 kB/s | 174 kB 00:00 2023-06-23T00:31:20.080 INFO:teuthology.orchestra.run.smithi064.stdout:(16/120): librados-devel-17.2.6-719.g42ddc46e.e 1.3 MB/s | 130 kB 00:00 2023-06-23T00:31:20.694 INFO:teuthology.orchestra.run.smithi064.stdout:(17/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 9.0 MB/s | 5.5 MB 00:00 2023-06-23T00:31:20.749 INFO:teuthology.orchestra.run.smithi064.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 823 kB/s | 45 kB 00:00 2023-06-23T00:31:20.808 INFO:teuthology.orchestra.run.smithi064.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 1.9 MB/s | 115 kB 00:00 2023-06-23T00:31:20.872 INFO:teuthology.orchestra.run.smithi064.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 2.9 MB/s | 190 kB 00:00 2023-06-23T00:31:20.912 INFO:teuthology.orchestra.run.smithi064.stdout:(21/120): libradosstriper1-17.2.6-719.g42ddc46e 547 kB/s | 506 kB 00:00 2023-06-23T00:31:20.945 INFO:teuthology.orchestra.run.smithi064.stdout:(22/120): python3-rados-17.2.6-719.g42ddc46e.el 5.0 MB/s | 371 kB 00:00 2023-06-23T00:31:21.009 INFO:teuthology.orchestra.run.smithi064.stdout:(23/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 1.6 MB/s | 107 kB 00:00 2023-06-23T00:31:21.075 INFO:teuthology.orchestra.run.smithi064.stdout:(24/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.3 MB/s | 91 kB 00:00 2023-06-23T00:31:21.454 INFO:teuthology.orchestra.run.smithi064.stdout:(25/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 658 kB/s | 357 kB 00:00 2023-06-23T00:31:21.477 INFO:teuthology.orchestra.run.smithi064.stdout:(26/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 8.6 MB/s | 3.5 MB 00:00 2023-06-23T00:31:21.540 INFO:teuthology.orchestra.run.smithi064.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 366 kB/s | 23 kB 00:00 2023-06-23T00:31:21.602 INFO:teuthology.orchestra.run.smithi064.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 1.9 MB/s | 121 kB 00:00 2023-06-23T00:31:21.640 INFO:teuthology.orchestra.run.smithi064.stdout:(29/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 931 kB/s | 173 kB 00:00 2023-06-23T00:31:21.827 INFO:teuthology.orchestra.run.smithi064.stdout:(30/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 7.6 MB/s | 1.7 MB 00:00 2023-06-23T00:31:21.893 INFO:teuthology.orchestra.run.smithi064.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 3.5 MB/s | 238 kB 00:00 2023-06-23T00:31:21.948 INFO:teuthology.orchestra.run.smithi064.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 872 kB/s | 47 kB 00:00 2023-06-23T00:31:21.998 INFO:teuthology.orchestra.run.smithi064.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 288 kB/s | 14 kB 00:00 2023-06-23T00:31:22.056 INFO:teuthology.orchestra.run.smithi064.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 4.5 MB/s | 264 kB 00:00 2023-06-23T00:31:22.111 INFO:teuthology.orchestra.run.smithi064.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.4 MB/s | 80 kB 00:00 2023-06-23T00:31:22.513 INFO:teuthology.orchestra.run.smithi064.stdout:(36/120): ceph-test-17.2.6-719.g42ddc46e.el8.x8 10 MB/s | 52 MB 00:04 2023-06-23T00:31:22.514 INFO:teuthology.orchestra.run.smithi064.stdout:(37/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 44 kB/s | 18 kB 00:00 2023-06-23T00:31:22.529 INFO:teuthology.orchestra.run.smithi064.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:31:22.558 INFO:teuthology.orchestra.run.smithi064.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:31:22.559 INFO:teuthology.orchestra.run.smithi064.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:31:22.560 INFO:teuthology.orchestra.run.smithi064.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:31:22.694 INFO:teuthology.orchestra.run.smithi064.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 328 kB/s | 59 kB 00:00 2023-06-23T00:31:22.694 INFO:teuthology.orchestra.run.smithi064.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:31:22.695 INFO:teuthology.orchestra.run.smithi064.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:31:22.767 INFO:teuthology.orchestra.run.smithi064.stdout:(39/120): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2023-06-23T00:31:22.768 INFO:teuthology.orchestra.run.smithi064.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm (IP: 8.43.85.72) 2023-06-23T00:31:22.778 INFO:teuthology.orchestra.run.smithi064.stdout:(40/120): libunwind-1.3.1-3.el8.x86_64.rpm 892 kB/s | 75 kB 00:00 2023-06-23T00:31:22.821 INFO:teuthology.orchestra.run.smithi064.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2023-06-23T00:31:22.840 INFO:teuthology.orchestra.run.smithi064.stdout:(42/120): python3-asyncssh-2.7.0-2.el8.noarch.r 6.4 MB/s | 476 kB 00:00 2023-06-23T00:31:22.885 INFO:teuthology.orchestra.run.smithi064.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 749 kB/s | 33 kB 00:00 2023-06-23T00:31:22.905 INFO:teuthology.orchestra.run.smithi064.stdout:(44/120): python3-beautifulsoup4-4.6.3-2.el8.1. 2.2 MB/s | 185 kB 00:00 2023-06-23T00:31:22.916 INFO:teuthology.orchestra.run.smithi064.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 531 kB/s | 16 kB 00:00 2023-06-23T00:31:22.961 INFO:teuthology.orchestra.run.smithi064.stdout:(46/120): python3-cheroot-8.5.2-1.el8.noarch.rp 3.3 MB/s | 173 kB 00:00 2023-06-23T00:31:22.961 INFO:teuthology.orchestra.run.smithi064.stdout:(47/120): python3-cherrypy-18.4.0-1.el8.noarch. 8.7 MB/s | 384 kB 00:00 2023-06-23T00:31:22.990 INFO:teuthology.orchestra.run.smithi064.stdout:(48/120): python3-jaraco-6.2-6.el8.noarch.rpm 364 kB/s | 11 kB 00:00 2023-06-23T00:31:22.995 INFO:teuthology.orchestra.run.smithi064.stdout:(49/120): python3-google-auth-1.1.1-10.el8.noar 2.1 MB/s | 81 kB 00:00 2023-06-23T00:31:23.021 INFO:teuthology.orchestra.run.smithi064.stdout:(50/120): python3-jaraco-functools-2.0-4.el8.no 571 kB/s | 18 kB 00:00 2023-06-23T00:31:23.053 INFO:teuthology.orchestra.run.smithi064.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 1.5 MB/s | 49 kB 00:00 2023-06-23T00:31:23.101 INFO:teuthology.orchestra.run.smithi064.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 1.2 MB/s | 59 kB 00:00 2023-06-23T00:31:23.105 INFO:teuthology.orchestra.run.smithi064.stdout:(53/120): python3-kubernetes-11.0.0-6.el8.noarc 11 MB/s | 1.2 MB 00:00 2023-06-23T00:31:23.134 INFO:teuthology.orchestra.run.smithi064.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 1.8 MB/s | 60 kB 00:00 2023-06-23T00:31:23.145 INFO:teuthology.orchestra.run.smithi064.stdout:(55/120): python3-pecan-1.3.2-9.el8.noarch.rpm 7.0 MB/s | 283 kB 00:00 2023-06-23T00:31:23.165 INFO:teuthology.orchestra.run.smithi064.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 505 kB/s | 16 kB 00:00 2023-06-23T00:31:23.176 INFO:teuthology.orchestra.run.smithi064.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 1.1 MB/s | 34 kB 00:00 2023-06-23T00:31:23.202 INFO:teuthology.orchestra.run.smithi064.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 5.2 MB/s | 196 kB 00:00 2023-06-23T00:31:23.208 INFO:teuthology.orchestra.run.smithi064.stdout:(59/120): python3-rsa-4.8-1.el8.noarch.rpm 1.9 MB/s | 61 kB 00:00 2023-06-23T00:31:23.234 INFO:teuthology.orchestra.run.smithi064.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 607 kB/s | 19 kB 00:00 2023-06-23T00:31:23.239 INFO:teuthology.orchestra.run.smithi064.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 781 kB/s | 24 kB 00:00 2023-06-23T00:31:23.265 INFO:teuthology.orchestra.run.smithi064.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 945 kB/s | 29 kB 00:00 2023-06-23T00:31:23.270 INFO:teuthology.orchestra.run.smithi064.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 891 kB/s | 27 kB 00:00 2023-06-23T00:31:23.305 INFO:teuthology.orchestra.run.smithi064.stdout:(64/120): python3-waitress-1.4.3-1.el8.noarch.r 6.3 MB/s | 255 kB 00:00 2023-06-23T00:31:23.306 INFO:teuthology.orchestra.run.smithi064.stdout:(65/120): python3-webob-1.8.5-1.el8.1.noarch.rp 6.9 MB/s | 251 kB 00:00 2023-06-23T00:31:23.339 INFO:teuthology.orchestra.run.smithi064.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 1.8 MB/s | 61 kB 00:00 2023-06-23T00:31:23.340 INFO:teuthology.orchestra.run.smithi064.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 2.4 MB/s | 85 kB 00:00 2023-06-23T00:31:23.370 INFO:teuthology.orchestra.run.smithi064.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 742 kB/s | 23 kB 00:00 2023-06-23T00:31:23.375 INFO:teuthology.orchestra.run.smithi064.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 15 MB/s | 70 kB 00:00 2023-06-23T00:31:23.427 INFO:teuthology.orchestra.run.smithi064.stdout:(70/120): thrift-0.13.0-2.el8.x86_64.rpm 20 MB/s | 1.7 MB 00:00 2023-06-23T00:31:23.429 INFO:teuthology.orchestra.run.smithi064.stdout:(71/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 348 kB/s | 19 kB 00:00 2023-06-23T00:31:23.461 INFO:teuthology.orchestra.run.smithi064.stdout:(72/120): python3-oauthlib-2.1.0-1.el8.noarch.r 4.8 MB/s | 155 kB 00:00 2023-06-23T00:31:23.493 INFO:teuthology.orchestra.run.smithi064.stdout:(73/120): python3-jwt-1.6.1-2.el8.noarch.rpm 656 kB/s | 43 kB 00:00 2023-06-23T00:31:23.507 INFO:teuthology.orchestra.run.smithi064.stdout:(74/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2023-06-23T00:31:23.527 INFO:teuthology.orchestra.run.smithi064.stdout:(75/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 1.4 MB/s | 48 kB 00:00 2023-06-23T00:31:23.534 INFO:teuthology.orchestra.run.smithi064.stdout:(76/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 8.0 MB/s | 229 kB 00:00 2023-06-23T00:31:23.560 INFO:teuthology.orchestra.run.smithi064.stdout:(77/120): python3-requests-oauthlib-1.0.0-1.el8 1.3 MB/s | 43 kB 00:00 2023-06-23T00:31:23.593 INFO:teuthology.orchestra.run.smithi064.stdout:(78/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 39 MB/s | 2.3 MB 00:00 2023-06-23T00:31:23.594 INFO:teuthology.orchestra.run.smithi064.stdout:(79/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 542 kB/s | 18 kB 00:00 2023-06-23T00:31:23.623 INFO:teuthology.orchestra.run.smithi064.stdout:(80/120): fuse-2.9.7-12.el8.x86_64.rpm 2.7 MB/s | 83 kB 00:00 2023-06-23T00:31:23.631 INFO:teuthology.orchestra.run.smithi064.stdout:(81/120): pcre2-devel-10.32-2.el8.x86_64.rpm 15 MB/s | 605 kB 00:00 2023-06-23T00:31:23.653 INFO:teuthology.orchestra.run.smithi064.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 1.9 MB/s | 59 kB 00:00 2023-06-23T00:31:23.663 INFO:teuthology.orchestra.run.smithi064.stdout:(83/120): libquadmath-8.4.1-1.el8.x86_64.rpm 5.2 MB/s | 169 kB 00:00 2023-06-23T00:31:23.683 INFO:teuthology.orchestra.run.smithi064.stdout:(84/120): libcom_err-devel-1.45.6-1.el8.x86_64. 1.2 MB/s | 38 kB 00:00 2023-06-23T00:31:23.697 INFO:teuthology.orchestra.run.smithi064.stdout:(85/120): libsepol-devel-2.9-2.el8.x86_64.rpm 2.5 MB/s | 86 kB 00:00 2023-06-23T00:31:23.715 INFO:teuthology.orchestra.run.smithi064.stdout:(86/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 1.7 MB/s | 58 kB 00:00 2023-06-23T00:31:23.730 INFO:teuthology.orchestra.run.smithi064.stdout:(87/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2023-06-23T00:31:23.754 INFO:teuthology.orchestra.run.smithi064.stdout:(88/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 14 MB/s | 559 kB 00:00 2023-06-23T00:31:23.769 INFO:teuthology.orchestra.run.smithi064.stdout:(89/120): libgfortran-8.4.1-1.el8.x86_64.rpm 16 MB/s | 639 kB 00:00 2023-06-23T00:31:23.789 INFO:teuthology.orchestra.run.smithi064.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 2023-06-23T00:31:23.800 INFO:teuthology.orchestra.run.smithi064.stdout:(91/120): libselinux-devel-2.9-5.el8.x86_64.rpm 6.5 MB/s | 200 kB 00:00 2023-06-23T00:31:23.817 INFO:teuthology.orchestra.run.smithi064.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 3.4 MB/s | 97 kB 00:00 2023-06-23T00:31:23.846 INFO:teuthology.orchestra.run.smithi064.stdout:(93/120): python3-prettytable-0.7.2-14.el8.noar 1.5 MB/s | 44 kB 00:00 2023-06-23T00:31:23.872 INFO:teuthology.orchestra.run.smithi064.stdout:(94/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.0 MB/s | 27 kB 00:00 2023-06-23T00:31:23.897 INFO:teuthology.orchestra.run.smithi064.stdout:(95/120): jq-1.5-12.el8.x86_64.rpm 6.1 MB/s | 161 kB 00:00 2023-06-23T00:31:23.925 INFO:teuthology.orchestra.run.smithi064.stdout:(96/120): flex-2.6.1-9.el8.x86_64.rpm 11 MB/s | 320 kB 00:00 2023-06-23T00:31:23.955 INFO:teuthology.orchestra.run.smithi064.stdout:(97/120): python3-werkzeug-0.12.2-4.el8.noarch. 15 MB/s | 457 kB 00:00 2023-06-23T00:31:24.093 INFO:teuthology.orchestra.run.smithi064.stdout:(98/120): python3-scipy-1.0.0-20.module+el8.1.0 49 MB/s | 15 MB 00:00 2023-06-23T00:31:24.120 INFO:teuthology.orchestra.run.smithi064.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 1.4 MB/s | 39 kB 00:00 2023-06-23T00:31:24.144 INFO:teuthology.orchestra.run.smithi064.stdout:(100/120): python3-babel-2.5.1-5.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2023-06-23T00:31:24.153 INFO:teuthology.orchestra.run.smithi064.stdout:(101/120): bison-3.0.4-10.el8.x86_64.rpm 21 MB/s | 688 kB 00:00 2023-06-23T00:31:24.174 INFO:teuthology.orchestra.run.smithi064.stdout:(102/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 8.6 MB/s | 259 kB 00:00 2023-06-23T00:31:24.183 INFO:teuthology.orchestra.run.smithi064.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 7.4 MB/s | 225 kB 00:00 2023-06-23T00:31:24.200 INFO:teuthology.orchestra.run.smithi064.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 4.7 MB/s | 126 kB 00:00 2023-06-23T00:31:24.230 INFO:teuthology.orchestra.run.smithi064.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 6.1 MB/s | 187 kB 00:00 2023-06-23T00:31:24.256 INFO:teuthology.orchestra.run.smithi064.stdout:(106/120): python3-numpy-1.14.3-9.el8.x86_64.rp 51 MB/s | 3.7 MB 00:00 2023-06-23T00:31:24.263 INFO:teuthology.orchestra.run.smithi064.stdout:(107/120): python3-jinja2-2.10.1-2.el8_0.noarch 16 MB/s | 538 kB 00:00 2023-06-23T00:31:24.284 INFO:teuthology.orchestra.run.smithi064.stdout:(108/120): socat-1.7.3.3-2.el8.x86_64.rpm 11 MB/s | 302 kB 00:00 2023-06-23T00:31:24.291 INFO:teuthology.orchestra.run.smithi064.stdout:(109/120): python3-mako-1.0.6-13.el8.noarch.rpm 5.6 MB/s | 157 kB 00:00 2023-06-23T00:31:24.313 INFO:teuthology.orchestra.run.smithi064.stdout:(110/120): python3-pyasn1-modules-0.3.7-6.el8.n 3.8 MB/s | 110 kB 00:00 2023-06-23T00:31:24.322 INFO:teuthology.orchestra.run.smithi064.stdout:(111/120): librdkafka-0.11.4-1.el8.x86_64.rpm 12 MB/s | 353 kB 00:00 2023-06-23T00:31:24.342 INFO:teuthology.orchestra.run.smithi064.stdout:(112/120): python3-jsonpointer-1.10-11.el8.noar 699 kB/s | 20 kB 00:00 2023-06-23T00:31:24.357 INFO:teuthology.orchestra.run.smithi064.stdout:(113/120): libtool-2.4.6-25.el8.x86_64.rpm 19 MB/s | 709 kB 00:00 2023-06-23T00:31:24.370 INFO:teuthology.orchestra.run.smithi064.stdout:(114/120): python3-pytz-2017.2-9.el8.noarch.rpm 1.9 MB/s | 54 kB 00:00 2023-06-23T00:31:24.462 INFO:teuthology.orchestra.run.smithi064.stdout:(115/120): openblas-threads-0.3.12-1.el8.x86_64 45 MB/s | 4.7 MB 00:00 2023-06-23T00:31:24.483 INFO:teuthology.orchestra.run.smithi064.stdout:(116/120): openblas-0.3.12-1.el8.x86_64.rpm 40 MB/s | 4.6 MB 00:00 2023-06-23T00:31:24.495 INFO:teuthology.orchestra.run.smithi064.stdout:(117/120): automake-1.16.1-7.el8.noarch.rpm 22 MB/s | 713 kB 00:00 2023-06-23T00:31:24.526 INFO:teuthology.orchestra.run.smithi064.stdout:(118/120): ceph-mgr-diskprediction-local-17.2.6 2.6 MB/s | 7.4 MB 00:02 2023-06-23T00:31:24.806 INFO:teuthology.orchestra.run.smithi064.stdout:(119/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 12 MB/s | 3.6 MB 00:00 2023-06-23T00:31:24.869 INFO:teuthology.orchestra.run.smithi064.stdout:(120/120): librados2-17.2.6-719.g42ddc46e.el8.x 9.0 MB/s | 3.5 MB 00:00 2023-06-23T00:31:24.873 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2023-06-23T00:31:24.873 INFO:teuthology.orchestra.run.smithi064.stdout:Total 22 MB/s | 201 MB 00:09 2023-06-23T00:31:24.926 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2023-06-23T00:31:25.051 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2023-06-23T00:31:25.051 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2023-06-23T00:31:26.800 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2023-06-23T00:31:26.805 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2023-06-23T00:31:31.764 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2023-06-23T00:31:32.025 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-23T00:31:32.203 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-23T00:31:32.374 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-23T00:31:32.561 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-23T00:31:32.992 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-23T00:31:33.177 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T00:31:33.605 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T00:31:34.128 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T00:31:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T00:31:34.452 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:31:34.569 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:31:34.775 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:31:35.085 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-06-23T00:31:35.780 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-23T00:31:35.943 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:31:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:31:36.415 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-23T00:31:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-23T00:31:36.819 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:31:36.937 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:31:37.252 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T00:31:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T00:31:37.541 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-06-23T00:31:37.783 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-06-23T00:31:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-06-23T00:31:38.159 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-06-23T00:31:38.318 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-23T00:31:38.458 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-23T00:31:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-23T00:31:39.779 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-23T00:31:40.330 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-23T00:31:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-06-23T00:31:41.320 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-06-23T00:31:41.607 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-06-23T00:31:41.848 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-06-23T00:31:42.132 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-23T00:31:42.333 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-23T00:31:42.462 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-23T00:31:42.751 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T00:31:42.976 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-06-23T00:31:43.307 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-06-23T00:31:43.534 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-23T00:31:45.726 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-23T00:31:46.087 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-06-23T00:31:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-06-23T00:31:47.455 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-06-23T00:31:47.689 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-06-23T00:31:47.935 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-23T00:31:48.106 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-23T00:31:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-23T00:31:48.489 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-23T00:31:48.825 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-23T00:31:49.066 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-06-23T00:31:49.250 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-06-23T00:31:49.466 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-06-23T00:31:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-06-23T00:31:49.850 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-23T00:31:49.936 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-23T00:31:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-06-23T00:31:50.256 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-06-23T00:31:50.474 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-06-23T00:31:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-06-23T00:31:50.871 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-06-23T00:31:51.020 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-06-23T00:31:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-06-23T00:31:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-06-23T00:31:51.547 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-06-23T00:31:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-06-23T00:31:52.096 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-06-23T00:31:52.367 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-06-23T00:31:52.653 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-06-23T00:31:52.825 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-23T00:31:53.022 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-23T00:31:53.268 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-23T00:31:53.459 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-23T00:31:53.665 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-23T00:31:53.821 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-23T00:31:53.945 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-23T00:31:54.161 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-23T00:31:54.406 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-23T00:31:54.621 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-23T00:31:54.861 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-23T00:31:55.015 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-23T00:31:55.299 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-23T00:31:55.580 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-23T00:31:55.756 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-23T00:31:56.096 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-23T00:31:56.358 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-23T00:31:56.536 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-23T00:31:56.790 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-23T00:31:58.008 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-23T00:31:58.333 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-23T00:31:58.586 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-23T00:31:58.932 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-23T00:31:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-23T00:31:59.556 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-23T00:31:59.793 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-23T00:32:00.018 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-23T00:32:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-23T00:32:05.588 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:32:06.499 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:32:07.136 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:32:07.386 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T00:32:07.458 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T00:32:07.632 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T00:32:25.853 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T00:32:25.854 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2023-06-23T00:32:25.854 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2023-06-23T00:32:25.854 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /dev 2023-06-23T00:32:25.854 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /run 2023-06-23T00:32:25.854 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2023-06-23T00:32:25.854 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2023-06-23T00:32:25.855 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2023-06-23T00:32:25.855 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2023-06-23T00:32:25.855 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /dev 2023-06-23T00:32:25.855 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:32:26.187 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T00:32:27.784 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T00:32:27.933 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T00:32:28.168 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T00:32:28.312 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T00:32:28.729 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T00:32:28.912 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T00:32:28.930 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T00:32:31.894 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T00:32:32.028 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T00:32:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-23T00:32:32.509 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-23T00:32:32.702 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T00:32:32.930 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T00:32:33.182 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T00:32:34.216 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T00:32:34.394 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T00:32:34.498 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T00:32:34.905 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-23T00:32:35.091 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T00:32:35.130 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T00:32:35.294 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T00:32:35.306 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T00:32:37.020 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-23T00:32:37.266 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T00:32:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T00:32:37.494 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T00:32:37.539 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T00:32:37.694 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T00:32:38.065 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T00:32:38.261 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T00:32:43.500 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T00:32:44.522 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-23T00:32:44.943 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-06-23T00:32:45.139 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 110/122 2023-06-23T00:32:45.369 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-06-23T00:32:45.542 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-06-23T00:32:45.555 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-06-23T00:32:45.739 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-23T00:32:45.872 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-23T00:32:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-23T00:32:46.242 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-23T00:32:46.479 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-23T00:32:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-06-23T00:32:46.698 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-06-23T00:32:46.892 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-06-23T00:32:46.902 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-06-23T00:32:47.020 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-06-23T00:32:47.020 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T00:32:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T00:32:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:32:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:32:47.780 INFO:teuthology.orchestra.run.smithi064.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-23T00:32:47.780 INFO:teuthology.orchestra.run.smithi064.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-23T00:32:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:32:48.094 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-23T00:32:48.095 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-23T00:32:48.095 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-23T00:32:48.095 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T00:32:48.095 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-23T00:32:48.095 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:32:48.095 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:32:48.096 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T00:32:48.096 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-23T00:32:48.096 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:32:48.096 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-23T00:32:48.096 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-23T00:32:48.096 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:32:48.096 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-23T00:32:48.097 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-23T00:32:48.097 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-23T00:32:48.097 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-23T00:32:48.097 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-23T00:32:48.097 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-23T00:32:48.097 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-23T00:32:48.097 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-23T00:32:48.097 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-23T00:32:48.098 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-23T00:32:48.098 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-23T00:32:48.098 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-23T00:32:48.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-23T00:32:48.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-23T00:32:48.104 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T00:32:48.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-23T00:32:48.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-23T00:32:48.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-23T00:32:48.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-23T00:32:48.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-23T00:32:48.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-23T00:32:48.105 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-23T00:32:48.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-23T00:32:48.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-23T00:32:48.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-23T00:32:48.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-23T00:32:48.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-23T00:32:48.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-23T00:32:48.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-23T00:32:48.106 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-23T00:32:48.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-23T00:32:48.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-23T00:32:48.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-23T00:32:48.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-23T00:32:48.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-23T00:32:48.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-23T00:32:48.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-23T00:32:48.107 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-23T00:32:48.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-23T00:32:48.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-23T00:32:48.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-23T00:32:48.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-23T00:32:48.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-23T00:32:48.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-23T00:32:48.108 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-23T00:32:48.109 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-23T00:32:48.109 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-23T00:32:48.109 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-23T00:32:48.109 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-23T00:32:48.109 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-23T00:32:48.109 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-23T00:32:48.109 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-23T00:32:48.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-23T00:32:48.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-23T00:32:48.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-23T00:32:48.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-23T00:32:48.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-23T00:32:48.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-23T00:32:48.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-06-23T00:32:48.110 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-06-23T00:32:48.111 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-06-23T00:32:48.111 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-06-23T00:32:48.111 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-06-23T00:32:48.111 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-06-23T00:32:48.111 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-06-23T00:32:48.111 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-06-23T00:32:48.111 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-06-23T00:32:48.111 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-06-23T00:32:48.112 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-06-23T00:32:48.112 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-06-23T00:32:48.112 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-06-23T00:32:48.112 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-06-23T00:32:48.112 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-06-23T00:32:48.112 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-06-23T00:32:48.112 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-06-23T00:32:48.112 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-06-23T00:32:48.113 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-06-23T00:32:48.113 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-06-23T00:32:48.113 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-06-23T00:32:48.113 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-06-23T00:32:48.113 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-06-23T00:32:48.113 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-06-23T00:32:48.113 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-06-23T00:32:48.114 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-06-23T00:32:48.114 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-06-23T00:32:48.114 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-06-23T00:32:48.114 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-06-23T00:32:48.114 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-06-23T00:32:48.114 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-06-23T00:32:48.114 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-06-23T00:32:48.114 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-06-23T00:32:48.115 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-06-23T00:32:48.115 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-06-23T00:32:48.115 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-06-23T00:32:48.115 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-06-23T00:32:48.115 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-06-23T00:32:48.115 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-06-23T00:32:48.115 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-06-23T00:32:48.115 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-06-23T00:32:48.116 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-06-23T00:32:48.116 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-06-23T00:32:48.116 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-06-23T00:32:48.116 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-06-23T00:32:48.116 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-06-23T00:32:48.116 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-06-23T00:32:48.116 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-23T00:32:48.117 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-23T00:32:48.117 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-23T00:32:50.355 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:32:50.355 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:44 ago on Fri 23 Jun 2023 12:31:06 AM UTC. 2023-06-23T00:32:52.499 INFO:teuthology.orchestra.run.smithi064.stdout:Installed products updated. 2023-06-23T00:32:52.658 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:32:52.658 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2023-06-23T00:32:52.659 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.659 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.659 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2023-06-23T00:32:52.659 INFO:teuthology.orchestra.run.smithi064.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T00:32:52.659 INFO:teuthology.orchestra.run.smithi064.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T00:32:52.659 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.659 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.659 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.660 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.660 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:32:52.660 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.660 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.660 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.661 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:32:52.661 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:32:52.661 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:32:52.661 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:32:52.661 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:32:52.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:32:52.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:32:52.663 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:32:52.663 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.663 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:32:52.663 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-23T00:32:52.663 INFO:teuthology.orchestra.run.smithi064.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T00:32:52.664 INFO:teuthology.orchestra.run.smithi064.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-23T00:32:52.664 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T00:32:52.664 INFO:teuthology.orchestra.run.smithi064.stdout: jq-1.5-12.el8.x86_64 2023-06-23T00:32:52.664 INFO:teuthology.orchestra.run.smithi064.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-23T00:32:52.664 INFO:teuthology.orchestra.run.smithi064.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-23T00:32:52.664 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T00:32:52.664 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.665 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.665 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.665 INFO:teuthology.orchestra.run.smithi064.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-23T00:32:52.665 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-23T00:32:52.665 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T00:32:52.665 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-23T00:32:52.665 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T00:32:52.666 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.666 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.666 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-23T00:32:52.666 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.666 INFO:teuthology.orchestra.run.smithi064.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-23T00:32:52.666 INFO:teuthology.orchestra.run.smithi064.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-23T00:32:52.667 INFO:teuthology.orchestra.run.smithi064.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T00:32:52.667 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T00:32:52.667 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-23T00:32:52.667 INFO:teuthology.orchestra.run.smithi064.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-23T00:32:52.667 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T00:32:52.667 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T00:32:52.667 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-23T00:32:52.668 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-23T00:32:52.668 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-23T00:32:52.668 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-23T00:32:52.668 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-23T00:32:52.668 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-23T00:32:52.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T00:32:52.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-23T00:32:52.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T00:32:52.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T00:32:52.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T00:32:52.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T00:32:52.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T00:32:52.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T00:32:52.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T00:32:52.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T00:32:52.671 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T00:32:52.671 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-23T00:32:52.671 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T00:32:52.672 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T00:32:52.672 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T00:32:52.672 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T00:32:52.672 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T00:32:52.673 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-23T00:32:52.673 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T00:32:52.673 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T00:32:52.674 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T00:32:52.674 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-23T00:32:52.674 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-23T00:32:52.674 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T00:32:52.675 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T00:32:52.675 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T00:32:52.675 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T00:32:52.676 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T00:32:52.676 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T00:32:52.676 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T00:32:52.677 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.677 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.677 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T00:32:52.677 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T00:32:52.678 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.678 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T00:32:52.678 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T00:32:52.679 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-23T00:32:52.679 INFO:teuthology.orchestra.run.smithi064.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T00:32:52.679 INFO:teuthology.orchestra.run.smithi064.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T00:32:52.679 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T00:32:52.680 INFO:teuthology.orchestra.run.smithi064.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T00:32:52.680 INFO:teuthology.orchestra.run.smithi064.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T00:32:52.680 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T00:32:52.680 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T00:32:52.680 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T00:32:52.680 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T00:32:52.680 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T00:32:52.681 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.681 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.681 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:32:52.681 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-23T00:32:52.681 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T00:32:52.681 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-23T00:32:52.682 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T00:32:52.682 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-23T00:32:52.682 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:32:52.682 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2023-06-23T00:32:52.923 DEBUG:teuthology.parallel:result is None 2023-06-23T00:37:26.750 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 41 kB/s | 16 MB 06:30 2023-06-23T00:37:30.003 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 4.9 MB/s | 24 kB 00:00 2023-06-23T00:37:30.380 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 126 MB/s | 38 MB 00:00 2023-06-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi086.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 118 MB/s | 33 MB 00:00 2023-06-23T00:37:43.861 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T00:37:43.862 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T00:37:43.872 INFO:teuthology.orchestra.run.smithi086.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-23T00:37:43.873 INFO:teuthology.orchestra.run.smithi086.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-23T00:37:43.874 INFO:teuthology.orchestra.run.smithi086.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-23T00:37:43.875 INFO:teuthology.orchestra.run.smithi086.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T00:37:43.876 INFO:teuthology.orchestra.run.smithi086.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T00:37:43.879 INFO:teuthology.orchestra.run.smithi086.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-23T00:37:43.880 INFO:teuthology.orchestra.run.smithi086.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-23T00:37:44.363 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2023-06-23T00:37:44.374 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================================================== 2023-06-23T00:37:44.374 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2023-06-23T00:37:44.374 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================================================== 2023-06-23T00:37:44.374 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2023-06-23T00:37:44.375 INFO:teuthology.orchestra.run.smithi086.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-23T00:37:44.375 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T00:37:44.375 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T00:37:44.375 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T00:37:44.375 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T00:37:44.375 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T00:37:44.375 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T00:37:44.375 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T00:37:44.376 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T00:37:44.376 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T00:37:44.376 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T00:37:44.376 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T00:37:44.376 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T00:37:44.376 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T00:37:44.377 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T00:37:44.377 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T00:37:44.377 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-23T00:37:44.377 INFO:teuthology.orchestra.run.smithi086.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-23T00:37:44.377 INFO:teuthology.orchestra.run.smithi086.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-23T00:37:44.377 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T00:37:44.377 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T00:37:44.377 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T00:37:44.378 INFO:teuthology.orchestra.run.smithi086.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-23T00:37:44.378 INFO:teuthology.orchestra.run.smithi086.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-23T00:37:44.378 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-23T00:37:44.378 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T00:37:44.378 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T00:37:44.378 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T00:37:44.378 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T00:37:44.379 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T00:37:44.379 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T00:37:44.379 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T00:37:44.379 INFO:teuthology.orchestra.run.smithi086.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-23T00:37:44.379 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2023-06-23T00:37:44.379 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T00:37:44.379 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T00:37:44.379 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2023-06-23T00:37:44.380 INFO:teuthology.orchestra.run.smithi086.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-23T00:37:44.380 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T00:37:44.380 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T00:37:44.380 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T00:37:44.380 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T00:37:44.380 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T00:37:44.380 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T00:37:44.381 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T00:37:44.381 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T00:37:44.381 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-23T00:37:44.381 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T00:37:44.381 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-23T00:37:44.381 INFO:teuthology.orchestra.run.smithi086.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-23T00:37:44.381 INFO:teuthology.orchestra.run.smithi086.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-23T00:37:44.381 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T00:37:44.382 INFO:teuthology.orchestra.run.smithi086.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-23T00:37:44.382 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-23T00:37:44.382 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T00:37:44.382 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-23T00:37:44.382 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-23T00:37:44.382 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T00:37:44.382 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-23T00:37:44.382 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T00:37:44.383 INFO:teuthology.orchestra.run.smithi086.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-23T00:37:44.383 INFO:teuthology.orchestra.run.smithi086.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-23T00:37:44.383 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T00:37:44.383 INFO:teuthology.orchestra.run.smithi086.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-23T00:37:44.383 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-23T00:37:44.383 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-23T00:37:44.383 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-23T00:37:44.384 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-23T00:37:44.384 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-23T00:37:44.384 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-23T00:37:44.384 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-23T00:37:44.384 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T00:37:44.384 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-23T00:37:44.384 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T00:37:44.384 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T00:37:44.385 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T00:37:44.385 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T00:37:44.385 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T00:37:44.385 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T00:37:44.385 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T00:37:44.385 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T00:37:44.385 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T00:37:44.386 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T00:37:44.386 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T00:37:44.386 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-23T00:37:44.386 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-23T00:37:44.386 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-23T00:37:44.386 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T00:37:44.386 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T00:37:44.386 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T00:37:44.387 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-23T00:37:44.387 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-23T00:37:44.387 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T00:37:44.387 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T00:37:44.387 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-23T00:37:44.387 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-23T00:37:44.387 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-23T00:37:44.387 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T00:37:44.388 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T00:37:44.388 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-23T00:37:44.388 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-23T00:37:44.388 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-23T00:37:44.388 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-23T00:37:44.388 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T00:37:44.388 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T00:37:44.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T00:37:44.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T00:37:44.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-23T00:37:44.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T00:37:44.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T00:37:44.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T00:37:44.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T00:37:44.390 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T00:37:44.390 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T00:37:44.390 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T00:37:44.390 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T00:37:44.390 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-23T00:37:44.390 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T00:37:44.390 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-23T00:37:44.390 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T00:37:44.391 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-23T00:37:44.391 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-23T00:37:44.391 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:37:44.391 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2023-06-23T00:37:44.391 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================================================== 2023-06-23T00:37:44.391 INFO:teuthology.orchestra.run.smithi086.stdout:Install 118 Packages 2023-06-23T00:37:44.391 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2023-06-23T00:37:44.391 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:37:44.392 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 201 M 2023-06-23T00:37:44.392 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2023-06-23T00:37:44.634 INFO:teuthology.orchestra.run.smithi086.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 35 kB/s | 6.4 kB 00:00 2023-06-23T00:37:44.876 INFO:teuthology.orchestra.run.smithi086.stdout:(2/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 12 MB/s | 5.2 MB 00:00 2023-06-23T00:37:44.898 INFO:teuthology.orchestra.run.smithi086.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 7.3 MB/s | 162 kB 00:00 2023-06-23T00:37:45.006 INFO:teuthology.orchestra.run.smithi086.stdout:(4/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 21 MB/s | 2.3 MB 00:00 2023-06-23T00:37:45.085 INFO:teuthology.orchestra.run.smithi086.stdout:(5/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 20 MB/s | 1.6 MB 00:00 2023-06-23T00:37:45.269 INFO:teuthology.orchestra.run.smithi086.stdout:(6/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 21 MB/s | 3.9 MB 00:00 2023-06-23T00:37:45.278 INFO:teuthology.orchestra.run.smithi086.stdout:(7/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 1.3 MB/s | 873 kB 00:00 2023-06-23T00:37:46.115 INFO:teuthology.orchestra.run.smithi086.stdout:(8/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86_ 20 MB/s | 17 MB 00:00 2023-06-23T00:37:46.137 INFO:teuthology.orchestra.run.smithi086.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 1.1 MB/s | 24 kB 00:00 2023-06-23T00:37:49.166 INFO:teuthology.orchestra.run.smithi086.stdout:(10/120): ceph-test-17.2.6-719.g42ddc46e.el8.x8 17 MB/s | 52 MB 00:03 2023-06-23T00:37:49.314 INFO:teuthology.orchestra.run.smithi086.stdout:(11/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 4.2 MB/s | 17 MB 00:04 2023-06-23T00:37:49.317 INFO:teuthology.orchestra.run.smithi086.stdout:(12/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 1.7 MB/s | 255 kB 00:00 2023-06-23T00:37:49.351 INFO:teuthology.orchestra.run.smithi086.stdout:(13/120): libcephfs-devel-17.2.6-719.g42ddc46e. 682 kB/s | 25 kB 00:00 2023-06-23T00:37:49.361 INFO:teuthology.orchestra.run.smithi086.stdout:(14/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 18 MB/s | 794 kB 00:00 2023-06-23T00:37:49.383 INFO:teuthology.orchestra.run.smithi086.stdout:(15/120): librados-devel-17.2.6-719.g42ddc46e.e 5.8 MB/s | 130 kB 00:00 2023-06-23T00:37:49.390 INFO:teuthology.orchestra.run.smithi086.stdout:(16/120): libcephsqlite-17.2.6-719.g42ddc46e.el 4.3 MB/s | 174 kB 00:00 2023-06-23T00:37:49.415 INFO:teuthology.orchestra.run.smithi086.stdout:(17/120): libradosstriper1-17.2.6-719.g42ddc46e 16 MB/s | 506 kB 00:00 2023-06-23T00:37:49.436 INFO:teuthology.orchestra.run.smithi086.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 2.1 MB/s | 45 kB 00:00 2023-06-23T00:37:49.458 INFO:teuthology.orchestra.run.smithi086.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 5.3 MB/s | 115 kB 00:00 2023-06-23T00:37:49.480 INFO:teuthology.orchestra.run.smithi086.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 8.4 MB/s | 190 kB 00:00 2023-06-23T00:37:49.514 INFO:teuthology.orchestra.run.smithi086.stdout:(21/120): python3-rados-17.2.6-719.g42ddc46e.el 11 MB/s | 371 kB 00:00 2023-06-23T00:37:49.553 INFO:teuthology.orchestra.run.smithi086.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 9.0 MB/s | 357 kB 00:00 2023-06-23T00:37:49.703 INFO:teuthology.orchestra.run.smithi086.stdout:(23/120): ceph-common-17.2.6-719.g42ddc46e.el8. 4.5 MB/s | 23 MB 00:05 2023-06-23T00:37:49.766 INFO:teuthology.orchestra.run.smithi086.stdout:(24/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 502 kB/s | 107 kB 00:00 2023-06-23T00:37:49.767 INFO:teuthology.orchestra.run.smithi086.stdout:(25/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.4 MB/s | 91 kB 00:00 2023-06-23T00:37:49.842 INFO:teuthology.orchestra.run.smithi086.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 2.2 MB/s | 173 kB 00:00 2023-06-23T00:37:49.907 INFO:teuthology.orchestra.run.smithi086.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 359 kB/s | 23 kB 00:00 2023-06-23T00:37:49.956 INFO:teuthology.orchestra.run.smithi086.stdout:(28/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 18 MB/s | 3.5 MB 00:00 2023-06-23T00:37:49.967 INFO:teuthology.orchestra.run.smithi086.stdout:(29/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 2.0 MB/s | 121 kB 00:00 2023-06-23T00:37:50.045 INFO:teuthology.orchestra.run.smithi086.stdout:(30/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 8.5 MB/s | 5.5 MB 00:00 2023-06-23T00:37:50.051 INFO:teuthology.orchestra.run.smithi086.stdout:(31/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 18 MB/s | 1.7 MB 00:00 2023-06-23T00:37:50.072 INFO:teuthology.orchestra.run.smithi086.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 2.2 MB/s | 47 kB 00:00 2023-06-23T00:37:50.086 INFO:teuthology.orchestra.run.smithi086.stdout:(33/120): ceph-mgr-modules-core-17.2.6-719.g42d 5.7 MB/s | 238 kB 00:00 2023-06-23T00:37:50.093 INFO:teuthology.orchestra.run.smithi086.stdout:(34/120): ceph-prometheus-alerts-17.2.6-719.g42 702 kB/s | 14 kB 00:00 2023-06-23T00:37:50.114 INFO:teuthology.orchestra.run.smithi086.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 3.7 MB/s | 80 kB 00:00 2023-06-23T00:37:50.125 INFO:teuthology.orchestra.run.smithi086.stdout:(36/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 6.7 MB/s | 264 kB 00:00 2023-06-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi086.stdout:(37/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 853 kB/s | 18 kB 00:00 2023-06-23T00:37:50.150 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:37:50.150 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:37:50.153 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:37:50.585 INFO:teuthology.orchestra.run.smithi086.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 131 kB/s | 59 kB 00:00 2023-06-23T00:37:50.586 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:37:50.588 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:37:50.729 INFO:teuthology.orchestra.run.smithi086.stdout:(39/120): ceph-mgr-diskprediction-local-17.2.6- 9.7 MB/s | 7.4 MB 00:00 2023-06-23T00:37:50.729 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm (IP: 8.43.85.72) 2023-06-23T00:37:50.770 INFO:teuthology.orchestra.run.smithi086.stdout:(40/120): libunwind-1.3.1-3.el8.x86_64.rpm 406 kB/s | 75 kB 00:00 2023-06-23T00:37:50.772 INFO:teuthology.orchestra.run.smithi086.stdout:(41/120): gperftools-libs-2.7-9.el8.x86_64.rpm 473 kB/s | 306 kB 00:00 2023-06-23T00:37:50.861 INFO:teuthology.orchestra.run.smithi086.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 2.0 MB/s | 185 kB 00:00 2023-06-23T00:37:50.946 INFO:teuthology.orchestra.run.smithi086.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 388 kB/s | 33 kB 00:00 2023-06-23T00:37:51.022 INFO:teuthology.orchestra.run.smithi086.stdout:(44/120): python3-asyncssh-2.7.0-2.el8.noarch.r 1.6 MB/s | 476 kB 00:00 2023-06-23T00:37:51.031 INFO:teuthology.orchestra.run.smithi086.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 193 kB/s | 16 kB 00:00 2023-06-23T00:37:51.108 INFO:teuthology.orchestra.run.smithi086.stdout:(46/120): python3-cheroot-8.5.2-1.el8.noarch.rp 2.0 MB/s | 173 kB 00:00 2023-06-23T00:37:51.119 INFO:teuthology.orchestra.run.smithi086.stdout:(47/120): python3-cherrypy-18.4.0-1.el8.noarch. 4.2 MB/s | 384 kB 00:00 2023-06-23T00:37:51.193 INFO:teuthology.orchestra.run.smithi086.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 957 kB/s | 81 kB 00:00 2023-06-23T00:37:51.196 INFO:teuthology.orchestra.run.smithi086.stdout:(49/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 102 kB/s | 44 kB 00:00 2023-06-23T00:37:51.204 INFO:teuthology.orchestra.run.smithi086.stdout:(50/120): python3-jaraco-6.2-6.el8.noarch.rpm 131 kB/s | 11 kB 00:00 2023-06-23T00:37:51.278 INFO:teuthology.orchestra.run.smithi086.stdout:(51/120): python3-jaraco-functools-2.0-4.el8.no 208 kB/s | 18 kB 00:00 2023-06-23T00:37:51.290 INFO:teuthology.orchestra.run.smithi086.stdout:(52/120): python3-logutils-0.3.5-11.el8.noarch. 574 kB/s | 49 kB 00:00 2023-06-23T00:37:51.363 INFO:teuthology.orchestra.run.smithi086.stdout:(53/120): python3-more-itertools-7.2.0-3.el8.no 694 kB/s | 59 kB 00:00 2023-06-23T00:37:51.374 INFO:teuthology.orchestra.run.smithi086.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 710 kB/s | 60 kB 00:00 2023-06-23T00:37:51.451 INFO:teuthology.orchestra.run.smithi086.stdout:(55/120): python3-pecan-1.3.2-9.el8.noarch.rpm 3.1 MB/s | 283 kB 00:00 2023-06-23T00:37:51.459 INFO:teuthology.orchestra.run.smithi086.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 184 kB/s | 16 kB 00:00 2023-06-23T00:37:51.537 INFO:teuthology.orchestra.run.smithi086.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 393 kB/s | 34 kB 00:00 2023-06-23T00:37:51.545 INFO:teuthology.orchestra.run.smithi086.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 2.2 MB/s | 196 kB 00:00 2023-06-23T00:37:51.621 INFO:teuthology.orchestra.run.smithi086.stdout:(59/120): python3-kubernetes-11.0.0-6.el8.noarc 2.7 MB/s | 1.2 MB 00:00 2023-06-23T00:37:51.622 INFO:teuthology.orchestra.run.smithi086.stdout:(60/120): python3-rsa-4.8-1.el8.noarch.rpm 719 kB/s | 61 kB 00:00 2023-06-23T00:37:51.630 INFO:teuthology.orchestra.run.smithi086.stdout:(61/120): python3-simplegeneric-0.8.1-17.el8.no 222 kB/s | 19 kB 00:00 2023-06-23T00:37:51.706 INFO:teuthology.orchestra.run.smithi086.stdout:(62/120): python3-singledispatch-3.4.0.3-18.el8 283 kB/s | 24 kB 00:00 2023-06-23T00:37:51.707 INFO:teuthology.orchestra.run.smithi086.stdout:(63/120): python3-tempora-1.14.1-5.el8.noarch.r 345 kB/s | 29 kB 00:00 2023-06-23T00:37:51.714 INFO:teuthology.orchestra.run.smithi086.stdout:(64/120): python3-trustme-0.6.0-4.el8.noarch.rp 326 kB/s | 27 kB 00:00 2023-06-23T00:37:51.793 INFO:teuthology.orchestra.run.smithi086.stdout:(65/120): python3-waitress-1.4.3-1.el8.noarch.r 2.9 MB/s | 255 kB 00:00 2023-06-23T00:37:51.796 INFO:teuthology.orchestra.run.smithi086.stdout:(66/120): python3-webob-1.8.5-1.el8.1.noarch.rp 2.8 MB/s | 251 kB 00:00 2023-06-23T00:37:51.799 INFO:teuthology.orchestra.run.smithi086.stdout:(67/120): python3-websocket-client-0.56.0-5.el8 722 kB/s | 61 kB 00:00 2023-06-23T00:37:51.878 INFO:teuthology.orchestra.run.smithi086.stdout:(68/120): python3-webtest-2.0.33-1.el8.noarch.r 1.0 MB/s | 85 kB 00:00 2023-06-23T00:37:51.880 INFO:teuthology.orchestra.run.smithi086.stdout:(69/120): python3-zc-lockfile-2.0-2.el8.noarch. 272 kB/s | 23 kB 00:00 2023-06-23T00:37:51.883 INFO:teuthology.orchestra.run.smithi086.stdout:(70/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 16 MB/s | 70 kB 00:00 2023-06-23T00:37:51.913 INFO:teuthology.orchestra.run.smithi086.stdout:(71/120): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2023-06-23T00:37:51.915 INFO:teuthology.orchestra.run.smithi086.stdout:(72/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 548 kB/s | 19 kB 00:00 2023-06-23T00:37:51.930 INFO:teuthology.orchestra.run.smithi086.stdout:(73/120): python3-oauthlib-2.1.0-1.el8.noarch.r 8.8 MB/s | 155 kB 00:00 2023-06-23T00:37:51.933 INFO:teuthology.orchestra.run.smithi086.stdout:(74/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 12 MB/s | 220 kB 00:00 2023-06-23T00:37:51.945 INFO:teuthology.orchestra.run.smithi086.stdout:(75/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 3.7 MB/s | 48 kB 00:00 2023-06-23T00:37:51.951 INFO:teuthology.orchestra.run.smithi086.stdout:(76/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 13 MB/s | 229 kB 00:00 2023-06-23T00:37:51.958 INFO:teuthology.orchestra.run.smithi086.stdout:(77/120): python3-requests-oauthlib-1.0.0-1.el8 2.8 MB/s | 43 kB 00:00 2023-06-23T00:37:51.971 INFO:teuthology.orchestra.run.smithi086.stdout:(78/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 1.4 MB/s | 18 kB 00:00 2023-06-23T00:37:51.981 INFO:teuthology.orchestra.run.smithi086.stdout:(79/120): thrift-0.13.0-2.el8.x86_64.rpm 9.6 MB/s | 1.7 MB 00:00 2023-06-23T00:37:52.002 INFO:teuthology.orchestra.run.smithi086.stdout:(80/120): pcre2-devel-10.32-2.el8.x86_64.rpm 20 MB/s | 605 kB 00:00 2023-06-23T00:37:52.012 INFO:teuthology.orchestra.run.smithi086.stdout:(81/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 37 MB/s | 2.3 MB 00:00 2023-06-23T00:37:52.013 INFO:teuthology.orchestra.run.smithi086.stdout:(82/120): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 83 kB 00:00 2023-06-23T00:37:52.017 INFO:teuthology.orchestra.run.smithi086.stdout:(83/120): elfutils-libelf-devel-0.182-3.el8.x86 3.7 MB/s | 59 kB 00:00 2023-06-23T00:37:52.029 INFO:teuthology.orchestra.run.smithi086.stdout:(84/120): libcom_err-devel-1.45.6-1.el8.x86_64. 2.4 MB/s | 38 kB 00:00 2023-06-23T00:37:52.031 INFO:teuthology.orchestra.run.smithi086.stdout:(85/120): libquadmath-8.4.1-1.el8.x86_64.rpm 8.8 MB/s | 169 kB 00:00 2023-06-23T00:37:52.036 INFO:teuthology.orchestra.run.smithi086.stdout:(86/120): libsepol-devel-2.9-2.el8.x86_64.rpm 5.6 MB/s | 86 kB 00:00 2023-06-23T00:37:52.046 INFO:teuthology.orchestra.run.smithi086.stdout:(87/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 3.5 MB/s | 58 kB 00:00 2023-06-23T00:37:52.048 INFO:teuthology.orchestra.run.smithi086.stdout:(88/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.8 MB/s | 47 kB 00:00 2023-06-23T00:37:52.054 INFO:teuthology.orchestra.run.smithi086.stdout:(89/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 26 MB/s | 559 kB 00:00 2023-06-23T00:37:52.061 INFO:teuthology.orchestra.run.smithi086.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 5.0 MB/s | 70 kB 00:00 2023-06-23T00:37:52.069 INFO:teuthology.orchestra.run.smithi086.stdout:(91/120): libgfortran-8.4.1-1.el8.x86_64.rpm 28 MB/s | 639 kB 00:00 2023-06-23T00:37:52.071 INFO:teuthology.orchestra.run.smithi086.stdout:(92/120): libselinux-devel-2.9-5.el8.x86_64.rpm 11 MB/s | 200 kB 00:00 2023-06-23T00:37:52.081 INFO:teuthology.orchestra.run.smithi086.stdout:(93/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 5.0 MB/s | 97 kB 00:00 2023-06-23T00:37:52.090 INFO:teuthology.orchestra.run.smithi086.stdout:(94/120): python3-prettytable-0.7.2-14.el8.noar 2.3 MB/s | 44 kB 00:00 2023-06-23T00:37:52.095 INFO:teuthology.orchestra.run.smithi086.stdout:(95/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.8 MB/s | 27 kB 00:00 2023-06-23T00:37:52.107 INFO:teuthology.orchestra.run.smithi086.stdout:(96/120): jq-1.5-12.el8.x86_64.rpm 9.8 MB/s | 161 kB 00:00 2023-06-23T00:37:52.115 INFO:teuthology.orchestra.run.smithi086.stdout:(97/120): flex-2.6.1-9.el8.x86_64.rpm 17 MB/s | 320 kB 00:00 2023-06-23T00:37:52.126 INFO:teuthology.orchestra.run.smithi086.stdout:(98/120): python3-werkzeug-0.12.2-4.el8.noarch. 23 MB/s | 457 kB 00:00 2023-06-23T00:37:52.140 INFO:teuthology.orchestra.run.smithi086.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 2.7 MB/s | 39 kB 00:00 2023-06-23T00:37:52.164 INFO:teuthology.orchestra.run.smithi086.stdout:(100/120): bison-3.0.4-10.el8.x86_64.rpm 29 MB/s | 688 kB 00:00 2023-06-23T00:37:52.180 INFO:teuthology.orchestra.run.smithi086.stdout:(101/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 16 MB/s | 259 kB 00:00 2023-06-23T00:37:52.201 INFO:teuthology.orchestra.run.smithi086.stdout:(102/120): python3-babel-2.5.1-5.el8.noarch.rpm 56 MB/s | 4.8 MB 00:00 2023-06-23T00:37:52.204 INFO:teuthology.orchestra.run.smithi086.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 9.4 MB/s | 225 kB 00:00 2023-06-23T00:37:52.216 INFO:teuthology.orchestra.run.smithi086.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 8.2 MB/s | 126 kB 00:00 2023-06-23T00:37:52.232 INFO:teuthology.orchestra.run.smithi086.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 12 MB/s | 187 kB 00:00 2023-06-23T00:37:52.252 INFO:teuthology.orchestra.run.smithi086.stdout:(106/120): python3-jinja2-2.10.1-2.el8_0.noarch 26 MB/s | 538 kB 00:00 2023-06-23T00:37:52.351 INFO:teuthology.orchestra.run.smithi086.stdout:(107/120): python3-scipy-1.0.0-20.module+el8.1. 52 MB/s | 15 MB 00:00 2023-06-23T00:37:52.355 INFO:teuthology.orchestra.run.smithi086.stdout:(108/120): socat-1.7.3.3-2.el8.x86_64.rpm 2.9 MB/s | 302 kB 00:00 2023-06-23T00:37:52.369 INFO:teuthology.orchestra.run.smithi086.stdout:(109/120): python3-numpy-1.14.3-9.el8.x86_64.rp 23 MB/s | 3.7 MB 00:00 2023-06-23T00:37:52.371 INFO:teuthology.orchestra.run.smithi086.stdout:(110/120): python3-mako-1.0.6-13.el8.noarch.rpm 7.7 MB/s | 157 kB 00:00 2023-06-23T00:37:52.372 INFO:teuthology.orchestra.run.smithi086.stdout:(111/120): python3-pyasn1-modules-0.3.7-6.el8.n 6.5 MB/s | 110 kB 00:00 2023-06-23T00:37:52.387 INFO:teuthology.orchestra.run.smithi086.stdout:(112/120): librdkafka-0.11.4-1.el8.x86_64.rpm 19 MB/s | 353 kB 00:00 2023-06-23T00:37:52.387 INFO:teuthology.orchestra.run.smithi086.stdout:(113/120): python3-jsonpointer-1.10-11.el8.noar 1.2 MB/s | 20 kB 00:00 2023-06-23T00:37:52.395 INFO:teuthology.orchestra.run.smithi086.stdout:(114/120): libtool-2.4.6-25.el8.x86_64.rpm 30 MB/s | 709 kB 00:00 2023-06-23T00:37:52.404 INFO:teuthology.orchestra.run.smithi086.stdout:(115/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.1 MB/s | 54 kB 00:00 2023-06-23T00:37:52.429 INFO:teuthology.orchestra.run.smithi086.stdout:(116/120): automake-1.16.1-7.el8.noarch.rpm 28 MB/s | 713 kB 00:00 2023-06-23T00:37:52.465 INFO:teuthology.orchestra.run.smithi086.stdout:(117/120): openblas-threads-0.3.12-1.el8.x86_64 61 MB/s | 4.7 MB 00:00 2023-06-23T00:37:52.484 INFO:teuthology.orchestra.run.smithi086.stdout:(118/120): openblas-0.3.12-1.el8.x86_64.rpm 51 MB/s | 4.6 MB 00:00 2023-06-23T00:37:52.665 INFO:teuthology.orchestra.run.smithi086.stdout:(119/120): librados2-17.2.6-719.g42ddc46e.el8.x 15 MB/s | 3.5 MB 00:00 2023-06-23T00:37:53.204 INFO:teuthology.orchestra.run.smithi086.stdout:(120/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 4.9 MB/s | 3.6 MB 00:00 2023-06-23T00:37:53.211 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2023-06-23T00:37:53.212 INFO:teuthology.orchestra.run.smithi086.stdout:Total 23 MB/s | 201 MB 00:08 2023-06-23T00:37:53.259 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2023-06-23T00:37:53.372 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2023-06-23T00:37:53.372 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2023-06-23T00:37:55.118 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2023-06-23T00:37:55.123 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2023-06-23T00:37:59.945 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2023-06-23T00:38:00.196 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-23T00:38:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-23T00:38:00.567 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-23T00:38:00.802 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-23T00:38:01.187 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-23T00:38:01.427 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T00:38:01.824 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T00:38:02.051 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T00:38:02.164 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T00:38:02.393 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:38:02.475 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:38:02.682 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:38:03.039 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-06-23T00:38:03.752 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-23T00:38:03.943 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:38:04.083 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:38:04.305 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-23T00:38:04.500 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-23T00:38:04.726 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:38:04.833 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:38:05.026 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T00:38:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T00:38:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-06-23T00:38:05.582 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-06-23T00:38:05.866 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-06-23T00:38:06.048 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-06-23T00:38:06.251 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-23T00:38:06.394 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-23T00:38:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-23T00:38:07.129 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-23T00:38:07.334 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-23T00:38:07.894 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-23T00:38:08.109 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-06-23T00:38:08.829 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-06-23T00:38:09.112 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-06-23T00:38:09.403 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-06-23T00:38:09.746 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-23T00:38:09.940 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-23T00:38:10.061 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-23T00:38:10.326 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T00:38:10.582 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-06-23T00:38:11.303 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-06-23T00:38:11.534 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-23T00:38:13.634 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-23T00:38:14.015 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-06-23T00:38:15.069 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-06-23T00:38:15.439 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-06-23T00:38:15.736 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-06-23T00:38:16.023 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-23T00:38:16.286 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-23T00:38:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-23T00:38:16.705 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-23T00:38:17.000 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-23T00:38:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-06-23T00:38:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-06-23T00:38:17.679 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-06-23T00:38:17.911 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-06-23T00:38:18.124 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-23T00:38:18.202 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-23T00:38:18.416 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-06-23T00:38:18.596 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-06-23T00:38:18.808 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-06-23T00:38:19.030 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-06-23T00:38:19.202 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-06-23T00:38:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-06-23T00:38:19.512 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-06-23T00:38:19.744 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-06-23T00:38:19.945 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-06-23T00:38:20.263 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-06-23T00:38:20.516 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-06-23T00:38:20.766 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-06-23T00:38:21.126 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-06-23T00:38:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-23T00:38:21.496 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-23T00:38:21.764 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-23T00:38:21.949 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-23T00:38:22.164 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-23T00:38:22.378 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-23T00:38:22.535 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-23T00:38:22.768 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-23T00:38:23.037 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-23T00:38:23.294 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-23T00:38:23.502 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-23T00:38:23.698 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-23T00:38:23.985 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-23T00:38:24.220 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-23T00:38:24.414 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-23T00:38:24.730 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-23T00:38:24.974 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-23T00:38:25.234 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-23T00:38:25.532 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-23T00:38:26.720 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-23T00:38:27.029 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-23T00:38:27.275 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-23T00:38:27.636 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-23T00:38:27.943 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-23T00:38:28.295 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-23T00:38:28.566 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-23T00:38:28.790 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-23T00:38:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-23T00:38:34.355 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:38:34.656 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:38:35.281 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:38:35.571 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T00:38:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T00:38:35.814 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T00:38:53.580 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T00:38:53.580 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2023-06-23T00:38:53.580 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2023-06-23T00:38:53.580 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2023-06-23T00:38:53.581 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /run 2023-06-23T00:38:53.581 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2023-06-23T00:38:53.581 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2023-06-23T00:38:53.581 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2023-06-23T00:38:53.581 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2023-06-23T00:38:53.581 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /dev 2023-06-23T00:38:53.581 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:38:53.995 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T00:38:55.474 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T00:38:55.676 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T00:38:55.898 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T00:38:56.045 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T00:38:56.434 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T00:38:56.636 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T00:38:56.653 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T00:38:59.422 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T00:38:59.674 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T00:38:59.843 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-23T00:39:00.050 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-23T00:39:00.277 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T00:39:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T00:39:00.751 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T00:39:01.729 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T00:39:01.886 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T00:39:01.984 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T00:39:02.380 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-23T00:39:02.601 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T00:39:02.637 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T00:39:02.793 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T00:39:02.805 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T00:39:04.453 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-23T00:39:04.734 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T00:39:04.826 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T00:39:05.035 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T00:39:05.079 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T00:39:05.310 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T00:39:05.658 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T00:39:05.893 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T00:39:10.855 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T00:39:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-23T00:39:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-06-23T00:39:12.429 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 110/122 2023-06-23T00:39:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-06-23T00:39:12.901 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-06-23T00:39:12.912 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-06-23T00:39:13.072 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-23T00:39:13.247 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-23T00:39:13.401 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-23T00:39:13.612 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-23T00:39:13.890 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-23T00:39:14.100 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-06-23T00:39:14.162 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-06-23T00:39:14.392 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-06-23T00:39:14.401 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-06-23T00:39:14.511 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-06-23T00:39:14.511 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T00:39:14.753 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T00:39:14.754 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:39:15.304 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:39:15.305 INFO:teuthology.orchestra.run.smithi086.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-23T00:39:15.305 INFO:teuthology.orchestra.run.smithi086.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-23T00:39:15.305 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-23T00:39:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-23T00:39:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-23T00:39:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T00:39:15.656 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-23T00:39:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:39:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:39:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T00:39:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-23T00:39:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:39:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-23T00:39:15.657 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-23T00:39:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:39:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-23T00:39:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-23T00:39:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-23T00:39:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-23T00:39:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-23T00:39:15.658 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-23T00:39:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-23T00:39:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-23T00:39:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-23T00:39:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-23T00:39:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-23T00:39:15.659 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-23T00:39:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-23T00:39:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-23T00:39:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T00:39:15.666 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-23T00:39:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-23T00:39:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-23T00:39:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-23T00:39:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-23T00:39:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-23T00:39:15.667 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-23T00:39:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-23T00:39:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-23T00:39:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-23T00:39:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-23T00:39:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-23T00:39:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-23T00:39:15.668 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-23T00:39:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-23T00:39:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-23T00:39:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-23T00:39:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-23T00:39:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-23T00:39:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-23T00:39:15.669 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-23T00:39:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-23T00:39:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-23T00:39:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-23T00:39:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-23T00:39:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-23T00:39:15.670 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-23T00:39:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-23T00:39:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-23T00:39:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-23T00:39:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-23T00:39:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-23T00:39:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-23T00:39:15.671 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-23T00:39:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-23T00:39:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-23T00:39:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-23T00:39:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-23T00:39:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-23T00:39:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-23T00:39:15.672 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-23T00:39:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-23T00:39:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-23T00:39:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-06-23T00:39:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-06-23T00:39:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-06-23T00:39:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-06-23T00:39:15.673 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-06-23T00:39:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-06-23T00:39:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-06-23T00:39:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-06-23T00:39:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-06-23T00:39:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-06-23T00:39:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-06-23T00:39:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-06-23T00:39:15.674 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-06-23T00:39:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-06-23T00:39:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-06-23T00:39:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-06-23T00:39:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-06-23T00:39:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-06-23T00:39:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-06-23T00:39:15.675 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-06-23T00:39:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-06-23T00:39:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-06-23T00:39:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-06-23T00:39:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-06-23T00:39:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-06-23T00:39:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-06-23T00:39:15.676 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-06-23T00:39:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-06-23T00:39:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-06-23T00:39:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-06-23T00:39:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-06-23T00:39:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-06-23T00:39:15.677 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-06-23T00:39:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-06-23T00:39:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-06-23T00:39:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-06-23T00:39:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-06-23T00:39:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-06-23T00:39:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-06-23T00:39:15.678 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-06-23T00:39:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-06-23T00:39:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-06-23T00:39:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-06-23T00:39:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-06-23T00:39:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-06-23T00:39:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-06-23T00:39:15.679 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-06-23T00:39:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-23T00:39:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-23T00:39:15.680 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-23T00:39:17.897 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:39:17.897 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:42 ago on Fri 23 Jun 2023 12:37:35 AM UTC. 2023-06-23T00:39:19.999 INFO:teuthology.orchestra.run.smithi086.stdout:Installed products updated. 2023-06-23T00:39:20.140 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:20.140 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2023-06-23T00:39:20.140 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.140 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.141 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2023-06-23T00:39:20.141 INFO:teuthology.orchestra.run.smithi086.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T00:39:20.141 INFO:teuthology.orchestra.run.smithi086.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T00:39:20.141 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.141 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.141 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.141 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:39:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:39:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:39:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:39:20.142 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:39:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:39:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:39:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.143 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.144 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:39:20.144 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:39:20.144 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.144 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:39:20.144 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-23T00:39:20.144 INFO:teuthology.orchestra.run.smithi086.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T00:39:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-23T00:39:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T00:39:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: jq-1.5-12.el8.x86_64 2023-06-23T00:39:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-23T00:39:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-23T00:39:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T00:39:20.145 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-23T00:39:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-23T00:39:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T00:39:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-23T00:39:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T00:39:20.146 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-23T00:39:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-23T00:39:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-23T00:39:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T00:39:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T00:39:20.147 INFO:teuthology.orchestra.run.smithi086.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-23T00:39:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-23T00:39:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T00:39:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T00:39:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-23T00:39:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-23T00:39:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-23T00:39:20.148 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-23T00:39:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-23T00:39:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-23T00:39:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T00:39:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-23T00:39:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T00:39:20.149 INFO:teuthology.orchestra.run.smithi086.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T00:39:20.150 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T00:39:20.150 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.150 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.150 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.150 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T00:39:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T00:39:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T00:39:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T00:39:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T00:39:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T00:39:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-23T00:39:20.151 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T00:39:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T00:39:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T00:39:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T00:39:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T00:39:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-23T00:39:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T00:39:20.152 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T00:39:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T00:39:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-23T00:39:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-23T00:39:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T00:39:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T00:39:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T00:39:20.153 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T00:39:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T00:39:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T00:39:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T00:39:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T00:39:20.154 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T00:39:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T00:39:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T00:39:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-23T00:39:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T00:39:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T00:39:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T00:39:20.155 INFO:teuthology.orchestra.run.smithi086.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T00:39:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T00:39:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T00:39:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T00:39:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T00:39:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T00:39:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T00:39:20.156 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.157 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.157 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:39:20.157 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-23T00:39:20.157 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T00:39:20.157 INFO:teuthology.orchestra.run.smithi086.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-23T00:39:20.157 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T00:39:20.158 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-23T00:39:20.158 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:20.158 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2023-06-23T00:39:20.367 DEBUG:teuthology.parallel:result is None 2023-06-23T00:39:20.367 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:39:20.584 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T00:39:20.611 INFO:teuthology.orchestra.run.smithi064.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T00:39:20.612 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T00:39:20.613 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T00:39:20.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:39:20.745 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T00:39:20.765 INFO:teuthology.orchestra.run.smithi086.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T00:39:20.766 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T00:39:20.766 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T00:39:20.766 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-06-23T00:39:20.767 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:20.767 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T00:39:20.812 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:20.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T00:39:20.853 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-06-23T00:39:20.853 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:20.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T00:39:20.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T00:39:20.949 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:20.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T00:39:20.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T00:39:21.038 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-06-23T00:39:21.039 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:21.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T00:39:21.066 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T00:39:21.133 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:21.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T00:39:21.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T00:39:21.224 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-06-23T00:39:21.225 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:21.225 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T00:39:21.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T00:39:21.315 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:21.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T00:39:21.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T00:39:21.404 INFO:teuthology.run_tasks:Running task ceph... 2023-06-23T00:39:21.481 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-06-23T00:39:21.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2023-06-23T00:39:21.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2023-06-23T00:39:21.510 INFO:tasks.ceph:Disabling ceph logrotate... 2023-06-23T00:39:21.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-23T00:39:21.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-23T00:39:21.574 INFO:tasks.ceph:Creating extra log directories... 2023-06-23T00:39:21.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-23T00:39:21.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-23T00:39:21.656 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-06-23T00:39:21.656 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-06-23T00:39:21.657 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-06-23T00:39:21.657 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:21.657 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-23T00:39:21.692 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-23T00:39:21.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-23T00:39:21.832 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:21.832 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-23T00:39:21.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-23T00:39:21.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-23T00:39:21.993 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-06-23T00:39:21.993 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'MDS_FAILED', 'MDS_DAMAGE'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-06-23T00:39:21.994 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876', 'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7311876', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'MDS_FAILED', 'MDS_DAMAGE'], 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 9037, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI35AffQKiJerBXQiexJsHNi/eRFM1jEN4uDgZt8gEo24uO9C7Qevh+1aUsEOcUiyFnWYA8rSQfZ+ukerdf7UXM=', 'smithi086.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5jJYX/k8e04S5zh43QvZfHdNrNLdUA9OK/jgcrj2wjYuUsSN+K3Oo5ocRKvqb4CqxrGPGLjG2JmlD44PlF3+NxFjcFM/JdCe/A3Cj+/w2m09YrQUPEio0hSfLNCYGdAGmVdD/1U3urI8Z0eWooe3gQCj6E3+Qkh8/gVa/JQUYIrnsvI7OU/ZMSXrbhmBKy164mYKMTQBClxOV4Yb9kPf4CeNWK5SF5KrYu7n3nGO2CThcRomrpdBl/W8h0ZEbD3ALzmANcLgfJCZZuKdUM7xtCelXLEtvEqtinRTSelX6Xo7f859ZOUcIn3V1jQ6dpG6GxnF58IrXQI4llkJWfo00MvtlMZ7ZMgh3RRA97j6M+I8LrHayFNDE2Rk6eCOgfnQ5BLvu93X7VVmSBmUBWCwEpHOaMSkPnH+lTZkvTdjnyn8capN1C8Rpnr6H/27RTccw20khijHhhdvrUh2oisPBw4vGLD1WdfW2bpTIQXQggaV3S/Cvy+rVKBPzp2ZIkTE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, {'ceph': None}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_recovery_pool']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '076bbebc42a14f7d568aaa78eabb0038327bcb23', 'timestamp': '2023-06-22_15:25:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-06-23T00:39:21.994 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-23T00:39:21.997 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-23T00:39:22.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2023-06-23T00:39:22.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2023-06-23T00:39:22.107 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:22.108 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T00:39:22.130 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T00:39:22.131 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2023-06-23T00:39:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T00:39:22.186 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T00:39:22.186 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 205633 Links: 1 2023-06-23T00:39:22.186 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T00:39:22.187 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T00:39:22.187 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-06-23 00:32:47.683946156 +0000 2023-06-23T00:39:22.187 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-06-23 00:30:18.402624372 +0000 2023-06-23T00:39:22.187 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-06-23 00:30:18.402624372 +0000 2023-06-23T00:39:22.187 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-06-23T00:39:22.187 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T00:39:22.251 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-06-23T00:39:22.251 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-06-23T00:39:22.251 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000332469 s, 1.5 MB/s 2023-06-23T00:39:22.253 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T00:39:22.309 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2023-06-23T00:39:22.364 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T00:39:22.364 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T00:39:22.364 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 208653 Links: 1 2023-06-23T00:39:22.365 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T00:39:22.365 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T00:39:22.365 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-06-23 00:32:47.683946156 +0000 2023-06-23T00:39:22.365 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-06-23 00:30:18.944613806 +0000 2023-06-23T00:39:22.365 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-06-23 00:30:18.944613806 +0000 2023-06-23T00:39:22.365 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-06-23T00:39:22.366 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T00:39:22.427 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-06-23T00:39:22.428 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-06-23T00:39:22.428 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000224198 s, 2.3 MB/s 2023-06-23T00:39:22.429 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T00:39:22.485 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2023-06-23T00:39:22.540 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T00:39:22.540 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T00:39:22.540 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 208773 Links: 1 2023-06-23T00:39:22.540 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T00:39:22.541 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T00:39:22.541 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-06-23 00:32:47.683946156 +0000 2023-06-23T00:39:22.541 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-06-23 00:30:19.470603551 +0000 2023-06-23T00:39:22.541 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-06-23 00:30:19.470603551 +0000 2023-06-23T00:39:22.541 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-06-23T00:39:22.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T00:39:22.604 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-06-23T00:39:22.604 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-06-23T00:39:22.604 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000222366 s, 2.3 MB/s 2023-06-23T00:39:22.605 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T00:39:22.660 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2023-06-23T00:39:22.716 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T00:39:22.716 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T00:39:22.716 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 209688 Links: 1 2023-06-23T00:39:22.717 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T00:39:22.717 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T00:39:22.717 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-06-23 00:32:47.684946137 +0000 2023-06-23T00:39:22.717 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-06-23 00:30:19.971593784 +0000 2023-06-23T00:39:22.717 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-06-23 00:30:19.971593784 +0000 2023-06-23T00:39:22.717 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-06-23T00:39:22.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T00:39:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-06-23T00:39:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-06-23T00:39:22.780 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000193662 s, 2.6 MB/s 2023-06-23T00:39:22.781 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T00:39:22.837 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T00:39:22.837 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:22.838 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T00:39:22.852 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T00:39:22.853 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2023-06-23T00:39:22.907 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T00:39:22.908 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T00:39:22.908 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 208557 Links: 1 2023-06-23T00:39:22.908 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T00:39:22.908 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T00:39:22.908 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-06-23 00:39:15.219404909 +0000 2023-06-23T00:39:22.908 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-06-23 00:27:57.158427455 +0000 2023-06-23T00:39:22.909 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-06-23 00:27:57.158427455 +0000 2023-06-23T00:39:22.909 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-06-23T00:39:22.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T00:39:22.971 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-06-23T00:39:22.971 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-06-23T00:39:22.971 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000227426 s, 2.3 MB/s 2023-06-23T00:39:22.972 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T00:39:23.028 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2023-06-23T00:39:23.082 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T00:39:23.083 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T00:39:23.083 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 210834 Links: 1 2023-06-23T00:39:23.083 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T00:39:23.083 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T00:39:23.083 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-06-23 00:39:15.219404909 +0000 2023-06-23T00:39:23.083 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-06-23 00:27:57.674416409 +0000 2023-06-23T00:39:23.084 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-06-23 00:27:57.674416409 +0000 2023-06-23T00:39:23.084 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-06-23T00:39:23.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T00:39:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-06-23T00:39:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-06-23T00:39:23.147 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000201175 s, 2.5 MB/s 2023-06-23T00:39:23.148 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T00:39:23.203 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2023-06-23T00:39:23.257 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T00:39:23.258 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T00:39:23.258 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 207703 Links: 1 2023-06-23T00:39:23.258 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T00:39:23.259 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T00:39:23.259 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-06-23 00:39:15.219404909 +0000 2023-06-23T00:39:23.259 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-06-23 00:27:58.182405538 +0000 2023-06-23T00:39:23.259 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-06-23 00:27:58.182405538 +0000 2023-06-23T00:39:23.259 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-06-23T00:39:23.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T00:39:23.321 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-06-23T00:39:23.321 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-06-23T00:39:23.322 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000224177 s, 2.3 MB/s 2023-06-23T00:39:23.323 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T00:39:23.378 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2023-06-23T00:39:23.433 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T00:39:23.434 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T00:39:23.434 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 212597 Links: 1 2023-06-23T00:39:23.434 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T00:39:23.434 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T00:39:23.434 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2023-06-23 00:39:15.219404909 +0000 2023-06-23T00:39:23.435 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2023-06-23 00:27:58.708394275 +0000 2023-06-23T00:39:23.435 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2023-06-23 00:27:58.708394275 +0000 2023-06-23T00:39:23.435 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2023-06-23T00:39:23.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T00:39:23.496 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2023-06-23T00:39:23.496 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2023-06-23T00:39:23.496 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000170852 s, 3.0 MB/s 2023-06-23T00:39:23.497 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T00:39:23.552 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T00:39:23.552 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-23T00:39:23.553 INFO:tasks.ceph:Generating config... 2023-06-23T00:39:23.554 INFO:tasks.ceph:[client] client mount timeout = 600 2023-06-23T00:39:23.554 INFO:tasks.ceph:[client] debug client = 20 2023-06-23T00:39:23.554 INFO:tasks.ceph:[client] debug ms = 1 2023-06-23T00:39:23.555 INFO:tasks.ceph:[client] pid file = 2023-06-23T00:39:23.555 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-06-23T00:39:23.555 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-06-23T00:39:23.555 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-06-23T00:39:23.555 INFO:tasks.ceph:[mds] debug mds = 20 2023-06-23T00:39:23.556 INFO:tasks.ceph:[mds] debug ms = 1 2023-06-23T00:39:23.556 INFO:tasks.ceph:[mds] mds debug frag = True 2023-06-23T00:39:23.556 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-06-23T00:39:23.556 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-06-23T00:39:23.556 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-06-23T00:39:23.557 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-06-23T00:39:23.557 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-06-23T00:39:23.557 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-06-23T00:39:23.557 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-06-23T00:39:23.557 INFO:tasks.ceph:[mgr] debug ms = 1 2023-06-23T00:39:23.558 INFO:tasks.ceph:[mon] debug mon = 20 2023-06-23T00:39:23.558 INFO:tasks.ceph:[mon] debug ms = 1 2023-06-23T00:39:23.558 INFO:tasks.ceph:[mon] debug paxos = 20 2023-06-23T00:39:23.558 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-06-23T00:39:23.558 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-06-23T00:39:23.559 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-06-23T00:39:23.559 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-06-23T00:39:23.559 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-06-23T00:39:23.559 INFO:tasks.ceph:[osd] debug ms = 1 2023-06-23T00:39:23.559 INFO:tasks.ceph:[osd] debug osd = 20 2023-06-23T00:39:23.560 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-06-23T00:39:23.560 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-06-23T00:39:23.560 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-06-23T00:39:23.560 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-06-23T00:39:23.560 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-06-23T00:39:23.561 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-06-23T00:39:23.561 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-06-23T00:39:23.561 INFO:tasks.ceph:Setting up mon.a... 2023-06-23T00:39:23.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-06-23T00:39:23.614 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.keyring 2023-06-23T00:39:23.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-06-23T00:39:23.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T00:39:23.723 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.64'), ('mon.b', '172.21.15.86'), ('mon.c', '[v2:172.21.15.86:3301,v1:172.21.15.86:6790]')] 2023-06-23T00:39:23.724 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.64,172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-06-23T00:39:23.725 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:23.725 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-06-23T00:39:23.778 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.64 --add b 172.21.15.86 --addv c '[v2:172.21.15.86:3301,v1:172.21.15.86:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:23.861 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:23.861 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid 72efde62-4c91-4277-87ee-356e50bd4cf5 2023-06-23T00:39:23.861 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = octopus 2023-06-23T00:39:23.861 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2023-06-23T00:39:23.862 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 72efde62-4c91-4277-87ee-356e50bd4cf5 2023-06-23T00:39:23.862 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2023-06-23T00:39:23.859024+0000 2023-06-23T00:39:23.862 INFO:teuthology.orchestra.run.smithi064.stdout:created 2023-06-23T00:39:23.859024+0000 2023-06-23T00:39:23.862 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 15 (octopus) 2023-06-23T00:39:23.862 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 1 2023-06-23T00:39:23.862 INFO:teuthology.orchestra.run.smithi064.stdout:0: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.a 2023-06-23T00:39:23.862 INFO:teuthology.orchestra.run.smithi064.stdout:1: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2023-06-23T00:39:23.862 INFO:teuthology.orchestra.run.smithi064.stdout:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2023-06-23T00:39:23.863 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-06-23T00:39:23.864 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-06-23T00:39:23.919 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 72efde62-4c91-4277-87ee-356e50bd4cf5... 2023-06-23T00:39:23.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-23T00:39:23.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-23T00:39:24.003 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2023-06-23T00:39:24.004 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2023-06-23T00:39:24.004 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-23T00:39:24.004 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2023-06-23T00:39:24.004 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.004 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2023-06-23T00:39:24.005 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.005 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2023-06-23T00:39:24.005 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.005 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2023-06-23T00:39:24.005 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2023-06-23T00:39:24.005 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.006 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2023-06-23T00:39:24.006 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2023-06-23T00:39:24.006 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.006 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-23T00:39:24.006 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2023-06-23T00:39:24.006 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.007 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2023-06-23T00:39:24.007 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-23T00:39:24.007 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.007 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2023-06-23T00:39:24.007 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.007 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2023-06-23T00:39:24.007 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2023-06-23T00:39:24.007 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2023-06-23T00:39:24.008 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2023-06-23T00:39:24.008 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2023-06-23T00:39:24.008 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2023-06-23T00:39:24.008 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2023-06-23T00:39:24.008 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-23T00:39:24.008 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-23T00:39:24.008 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2023-06-23T00:39:24.009 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.009 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-23T00:39:24.009 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.009 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2023-06-23T00:39:24.009 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.009 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2023-06-23T00:39:24.009 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.010 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2023-06-23T00:39:24.010 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2023-06-23T00:39:24.010 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2023-06-23T00:39:24.010 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "172.21.15.64,172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790]" 2023-06-23T00:39:24.010 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on skipped message = False 2023-06-23T00:39:24.010 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 72efde62-4c91-4277-87ee-356e50bd4cf5 2023-06-23T00:39:24.010 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.010 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2023-06-23T00:39:24.011 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2023-06-23T00:39:24.011 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.011 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2023-06-23T00:39:24.011 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2023-06-23T00:39:24.011 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2023-06-23T00:39:24.011 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.011 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2023-06-23T00:39:24.012 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2023-06-23T00:39:24.012 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.012 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2023-06-23T00:39:24.012 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2023-06-23T00:39:24.012 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2023-06-23T00:39:24.012 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.012 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2023-06-23T00:39:24.012 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2023-06-23T00:39:24.013 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.013 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2023-06-23T00:39:24.013 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.013 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2023-06-23T00:39:24.013 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.013 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2023-06-23T00:39:24.013 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.014 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2023-06-23T00:39:24.014 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2023-06-23T00:39:24.014 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.014 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2023-06-23T00:39:24.014 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2023-06-23T00:39:24.014 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2023-06-23T00:39:24.015 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2023-06-23T00:39:24.015 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 20 2023-06-23T00:39:24.015 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 20 2023-06-23T00:39:24.015 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-06-23T00:39:24.016 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2023-06-23T00:39:24.016 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2023-06-23T00:39:24.016 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2023-06-23T00:39:24.017 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2023-06-23T00:39:24.017 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2023-06-23T00:39:24.017 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2023-06-23T00:39:24.017 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2023-06-23T00:39:24.018 INFO:teuthology.orchestra.run.smithi064.stdout: osd op complaint time = 180 2023-06-23T00:39:24.018 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.018 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2023-06-23T00:39:24.019 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-06-23T00:39:24.019 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2023-06-23T00:39:24.019 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2023-06-23T00:39:24.019 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2023-06-23T00:39:24.020 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2023-06-23T00:39:24.020 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2023-06-23T00:39:24.020 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2023-06-23T00:39:24.020 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.021 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2023-06-23T00:39:24.021 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-06-23T00:39:24.021 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2023-06-23T00:39:24.021 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2023-06-23T00:39:24.021 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2023-06-23T00:39:24.021 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2023-06-23T00:39:24.021 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2023-06-23T00:39:24.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2023-06-23T00:39:24.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2023-06-23T00:39:24.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2023-06-23T00:39:24.022 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2023-06-23T00:39:24.022 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.022 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-23T00:39:24.022 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2023-06-23T00:39:24.023 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2023-06-23T00:39:24.023 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.023 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2023-06-23T00:39:24.023 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-23T00:39:24.024 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-23T00:39:24.024 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.024 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2023-06-23T00:39:24.024 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2023-06-23T00:39:24.024 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.025 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2023-06-23T00:39:24.025 INFO:teuthology.orchestra.run.smithi064.stdout: mon op complaint time = 120 2023-06-23T00:39:24.025 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:39:24.025 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2023-06-23T00:39:24.025 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2023-06-23T00:39:24.025 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2023-06-23T00:39:24.025 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2023-06-23T00:39:24.025 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-23T00:39:24.026 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-23T00:39:24.026 INFO:teuthology.orchestra.run.smithi064.stdout: client mount timeout = 600 2023-06-23T00:39:24.026 INFO:teuthology.orchestra.run.smithi064.stdout: debug client = 20 2023-06-23T00:39:24.026 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-06-23T00:39:24.026 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = "" 2023-06-23T00:39:24.026 INFO:teuthology.orchestra.run.smithi064.stdout: rados mon op timeout = 900 2023-06-23T00:39:24.026 INFO:teuthology.orchestra.run.smithi064.stdout: rados osd op timeout = 900 2023-06-23T00:39:24.027 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2023-06-23T00:39:24.027 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2023-06-23T00:39:24.027 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2023-06-23T00:39:24.027 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.a] 2023-06-23T00:39:24.027 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.c] 2023-06-23T00:39:24.027 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.b] 2023-06-23T00:39:24.027 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.d] 2023-06-23T00:39:24.027 INFO:teuthology.orchestra.run.smithi064.stdout:[mds] 2023-06-23T00:39:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: debug mds = 20 2023-06-23T00:39:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2023-06-23T00:39:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: mds debug frag = True 2023-06-23T00:39:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: mds debug scatterstat = True 2023-06-23T00:39:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: mds op complaint time = 180 2023-06-23T00:39:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: mds verify scatter = True 2023-06-23T00:39:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: osd op complaint time = 180 2023-06-23T00:39:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: rados mon op timeout = 900 2023-06-23T00:39:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: rados osd op timeout = 900 2023-06-23T00:39:24.030 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2023-06-23T00:39:24.030 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2023-06-23T00:39:24.030 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-23T00:39:24.030 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2023-06-23T00:39:24.031 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.031 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2023-06-23T00:39:24.031 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.031 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2023-06-23T00:39:24.031 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.031 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2023-06-23T00:39:24.031 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2023-06-23T00:39:24.032 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.032 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2023-06-23T00:39:24.032 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2023-06-23T00:39:24.032 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.032 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-23T00:39:24.032 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2023-06-23T00:39:24.033 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.033 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2023-06-23T00:39:24.034 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-23T00:39:24.034 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.034 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2023-06-23T00:39:24.035 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.035 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2023-06-23T00:39:24.035 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2023-06-23T00:39:24.035 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2023-06-23T00:39:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2023-06-23T00:39:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2023-06-23T00:39:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2023-06-23T00:39:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2023-06-23T00:39:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-23T00:39:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-23T00:39:24.036 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2023-06-23T00:39:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-23T00:39:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2023-06-23T00:39:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2023-06-23T00:39:24.037 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2023-06-23T00:39:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2023-06-23T00:39:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2023-06-23T00:39:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = "172.21.15.64,172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790]" 2023-06-23T00:39:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on skipped message = False 2023-06-23T00:39:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = 72efde62-4c91-4277-87ee-356e50bd4cf5 2023-06-23T00:39:24.038 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.038 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2023-06-23T00:39:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2023-06-23T00:39:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2023-06-23T00:39:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2023-06-23T00:39:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock profile = high_recovery_ops 2023-06-23T00:39:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.039 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2023-06-23T00:39:24.040 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2023-06-23T00:39:24.040 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.040 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2023-06-23T00:39:24.040 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2023-06-23T00:39:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2023-06-23T00:39:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2023-06-23T00:39:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2023-06-23T00:39:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2023-06-23T00:39:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.041 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2023-06-23T00:39:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2023-06-23T00:39:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2023-06-23T00:39:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2023-06-23T00:39:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2023-06-23T00:39:24.042 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2023-06-23T00:39:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore block size = 96636764160 2023-06-23T00:39:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore fsck on mount = True 2023-06-23T00:39:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluefs = 20 2023-06-23T00:39:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluestore = 20 2023-06-23T00:39:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-06-23T00:39:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2023-06-23T00:39:24.043 INFO:teuthology.orchestra.run.smithi086.stdout: debug rocksdb = 10 2023-06-23T00:39:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd backfillfull_ratio = 0.85 2023-06-23T00:39:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd full ratio = 0.9 2023-06-23T00:39:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd nearfull ratio = 0.8 2023-06-23T00:39:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: osd failsafe full ratio = 0.95 2023-06-23T00:39:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: osd objectstore = bluestore 2023-06-23T00:39:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: osd op complaint time = 180 2023-06-23T00:39:24.044 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.045 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2023-06-23T00:39:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-06-23T00:39:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2023-06-23T00:39:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2023-06-23T00:39:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2023-06-23T00:39:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2023-06-23T00:39:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2023-06-23T00:39:24.045 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2023-06-23T00:39:24.046 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.046 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2023-06-23T00:39:24.046 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-06-23T00:39:24.046 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2023-06-23T00:39:24.046 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2023-06-23T00:39:24.046 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2023-06-23T00:39:24.046 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2023-06-23T00:39:24.047 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2023-06-23T00:39:24.047 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2023-06-23T00:39:24.047 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2023-06-23T00:39:24.048 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2023-06-23T00:39:24.048 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2023-06-23T00:39:24.048 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-23T00:39:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: auth mon ticket ttl = 660 # 11m 2023-06-23T00:39:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: auth service ticket ttl = 240 # 4m 2023-06-23T00:39:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: # don't complain about insecure global_id in the test suite 2023-06-23T00:39:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-23T00:39:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-23T00:39:24.049 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.050 INFO:teuthology.orchestra.run.smithi086.stdout: # 1m isn't quite enough 2023-06-23T00:39:24.050 INFO:teuthology.orchestra.run.smithi086.stdout: mon_down_mkfs_grace = 2m 2023-06-23T00:39:24.050 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.050 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_filestore_osds = false 2023-06-23T00:39:24.050 INFO:teuthology.orchestra.run.smithi086.stdout: mon op complaint time = 120 2023-06-23T00:39:24.050 INFO:teuthology.orchestra.run.smithi086.stdout: 2023-06-23T00:39:24.050 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2023-06-23T00:39:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2023-06-23T00:39:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2023-06-23T00:39:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2023-06-23T00:39:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-23T00:39:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-23T00:39:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: client mount timeout = 600 2023-06-23T00:39:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: debug client = 20 2023-06-23T00:39:24.051 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-06-23T00:39:24.052 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = "" 2023-06-23T00:39:24.052 INFO:teuthology.orchestra.run.smithi086.stdout: rados mon op timeout = 900 2023-06-23T00:39:24.052 INFO:teuthology.orchestra.run.smithi086.stdout: rados osd op timeout = 900 2023-06-23T00:39:24.052 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2023-06-23T00:39:24.052 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.b] 2023-06-23T00:39:24.052 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.c] 2023-06-23T00:39:24.053 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.a] 2023-06-23T00:39:24.053 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.c] 2023-06-23T00:39:24.053 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.b] 2023-06-23T00:39:24.053 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.d] 2023-06-23T00:39:24.053 INFO:teuthology.orchestra.run.smithi086.stdout:[mds] 2023-06-23T00:39:24.053 INFO:teuthology.orchestra.run.smithi086.stdout: debug mds = 20 2023-06-23T00:39:24.054 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2023-06-23T00:39:24.054 INFO:teuthology.orchestra.run.smithi086.stdout: mds debug frag = True 2023-06-23T00:39:24.054 INFO:teuthology.orchestra.run.smithi086.stdout: mds debug scatterstat = True 2023-06-23T00:39:24.054 INFO:teuthology.orchestra.run.smithi086.stdout: mds op complaint time = 180 2023-06-23T00:39:24.054 INFO:teuthology.orchestra.run.smithi086.stdout: mds verify scatter = True 2023-06-23T00:39:24.054 INFO:teuthology.orchestra.run.smithi086.stdout: osd op complaint time = 180 2023-06-23T00:39:24.055 INFO:teuthology.orchestra.run.smithi086.stdout: rados mon op timeout = 900 2023-06-23T00:39:24.055 INFO:teuthology.orchestra.run.smithi086.stdout: rados osd op timeout = 900 2023-06-23T00:39:24.055 INFO:tasks.ceph:Creating admin key on mon.a... 2023-06-23T00:39:24.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-06-23T00:39:24.122 INFO:tasks.ceph:Copying monmap to all nodes... 2023-06-23T00:39:24.123 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:24.123 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-06-23T00:39:24.138 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:24.139 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-06-23T00:39:24.193 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:39:24.194 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:24.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-23T00:39:24.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T00:39:24.269 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:24.270 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:24.324 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2023-06-23T00:39:24.324 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:24.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-23T00:39:24.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T00:39:24.360 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:24.360 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:24.415 INFO:tasks.ceph:Setting up mon nodes... 2023-06-23T00:39:24.415 INFO:tasks.ceph:Setting up mgr nodes... 2023-06-23T00:39:24.416 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-06-23T00:39:24.465 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-06-23T00:39:24.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-06-23T00:39:24.576 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-06-23T00:39:24.579 INFO:tasks.ceph:Setting up mds nodes... 2023-06-23T00:39:24.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-06-23T00:39:24.629 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-06-23T00:39:24.631 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-06-23T00:39:24.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-06-23T00:39:24.746 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-06-23T00:39:24.748 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-06-23T00:39:24.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-06-23T00:39:24.821 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-06-23T00:39:24.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-06-23T00:39:24.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-06-23T00:39:24.939 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-06-23T00:39:24.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-06-23T00:39:24.965 INFO:tasks.ceph_client:Setting up client nodes... 2023-06-23T00:39:24.966 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-06-23T00:39:25.005 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-06-23T00:39:25.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-06-23T00:39:25.097 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-06-23T00:39:25.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-06-23T00:39:25.174 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-06-23T00:39:25.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-06-23T00:39:25.266 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-06-23T00:39:25.279 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-06-23T00:39:25.279 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-23T00:39:25.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-06-23T00:39:25.306 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T00:39:25.306 INFO:tasks.ceph:role: osd.0 2023-06-23T00:39:25.306 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:39:25.306 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-23T00:39:25.378 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T00:39:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T00:39:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T00:39:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 2023-06-23T00:39:25.379 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T00:39:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-06-23T00:39:25.379 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T00:39:25.379 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T00:39:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T00:39:25.379 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T00:39:25.809 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2023-06-23T00:39:25.810 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-06-23T00:39:25.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-06-23T00:39:25.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-06-23T00:39:26.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-06-23T00:39:26.087 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T00:39:26.087 INFO:tasks.ceph:role: osd.1 2023-06-23T00:39:26.088 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:39:26.088 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-23T00:39:26.159 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T00:39:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T00:39:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T00:39:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 2023-06-23T00:39:26.160 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T00:39:26.160 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-06-23T00:39:26.161 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T00:39:26.161 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T00:39:26.161 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T00:39:26.161 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T00:39:26.593 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2023-06-23T00:39:26.594 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-06-23T00:39:26.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-06-23T00:39:26.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-06-23T00:39:26.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-06-23T00:39:26.796 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T00:39:26.796 INFO:tasks.ceph:role: osd.2 2023-06-23T00:39:26.797 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:39:26.797 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-23T00:39:26.868 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T00:39:26.869 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T00:39:26.869 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T00:39:26.869 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 2023-06-23T00:39:26.869 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T00:39:26.869 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-06-23T00:39:26.870 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T00:39:26.870 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T00:39:26.870 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T00:39:26.870 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T00:39:27.316 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2023-06-23T00:39:27.318 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-06-23T00:39:27.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-06-23T00:39:27.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-06-23T00:39:27.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-06-23T00:39:27.597 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T00:39:27.597 INFO:tasks.ceph:role: osd.3 2023-06-23T00:39:27.597 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:39:27.598 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-23T00:39:27.669 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T00:39:27.669 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T00:39:27.669 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T00:39:27.669 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 2023-06-23T00:39:27.670 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T00:39:27.670 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2023-06-23T00:39:27.670 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T00:39:27.670 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T00:39:27.670 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T00:39:27.670 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T00:39:28.106 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2023-06-23T00:39:28.107 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2023-06-23T00:39:28.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-06-23T00:39:28.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-06-23T00:39:28.321 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:28.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:28.409+0000 7f59be0ea540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-06-23T00:39:28.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:28.409+0000 7f59be0ea540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-06-23T00:39:28.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:28.409+0000 7f59be0ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T00:39:28.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:28.409+0000 7f59be0ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T00:39:28.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:28.409+0000 7f59be0ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T00:39:28.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:28.410+0000 7f59be0ea540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-06-23T00:39:28.933 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:28.931+0000 7f59be0ea540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:29.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:29.692+0000 7f59be0ea540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:30.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:30.458+0000 7f59be0ea540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:31.255 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-06-23T00:39:31.282 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:31.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:31.362+0000 7f520e80c540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-06-23T00:39:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:31.362+0000 7f520e80c540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-06-23T00:39:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:31.362+0000 7f520e80c540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T00:39:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:31.362+0000 7f520e80c540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T00:39:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:31.362+0000 7f520e80c540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T00:39:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:31.363+0000 7f520e80c540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-06-23T00:39:31.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:31.887+0000 7f520e80c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:32.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:32.648+0000 7f520e80c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:33.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:33.411+0000 7f520e80c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:34.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-06-23T00:39:34.276 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:34.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:34.357+0000 7fe4d6e96540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-06-23T00:39:34.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:34.357+0000 7fe4d6e96540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-06-23T00:39:34.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:34.357+0000 7fe4d6e96540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T00:39:34.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:34.357+0000 7fe4d6e96540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T00:39:34.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:34.357+0000 7fe4d6e96540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T00:39:34.362 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:34.358+0000 7fe4d6e96540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-06-23T00:39:34.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:34.882+0000 7fe4d6e96540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:35.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:35.642+0000 7fe4d6e96540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:36.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:36.405+0000 7fe4d6e96540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:37.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-06-23T00:39:37.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:37.360 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:37.356+0000 7fcb49ce7540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-06-23T00:39:37.360 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:37.357+0000 7fcb49ce7540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-06-23T00:39:37.360 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:37.357+0000 7fcb49ce7540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T00:39:37.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:37.357+0000 7fcb49ce7540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T00:39:37.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:37.357+0000 7fcb49ce7540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T00:39:37.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:37.357+0000 7fcb49ce7540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-06-23T00:39:37.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:37.879+0000 7fcb49ce7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:38.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:38.639+0000 7fcb49ce7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:39.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:39.402+0000 7fcb49ce7540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:40.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-06-23T00:39:40.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-06-23T00:39:40.256 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T00:39:40.256 INFO:tasks.ceph:role: osd.4 2023-06-23T00:39:40.256 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2023-06-23T00:39:40.257 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-23T00:39:40.327 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T00:39:40.327 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T00:39:40.327 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T00:39:40.328 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 2023-06-23T00:39:40.328 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T00:39:40.328 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-06-23T00:39:40.328 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T00:39:40.328 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T00:39:40.328 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T00:39:40.328 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T00:39:40.755 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-06-23T00:39:40.757 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-06-23T00:39:40.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-06-23T00:39:40.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-06-23T00:39:40.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-06-23T00:39:41.008 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T00:39:41.008 INFO:tasks.ceph:role: osd.5 2023-06-23T00:39:41.009 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2023-06-23T00:39:41.009 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-23T00:39:41.080 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T00:39:41.080 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T00:39:41.080 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T00:39:41.080 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 2023-06-23T00:39:41.081 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T00:39:41.081 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-06-23T00:39:41.081 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T00:39:41.081 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T00:39:41.081 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T00:39:41.081 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T00:39:41.498 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-06-23T00:39:41.500 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-06-23T00:39:41.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-06-23T00:39:41.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-06-23T00:39:41.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-06-23T00:39:41.713 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T00:39:41.714 INFO:tasks.ceph:role: osd.6 2023-06-23T00:39:41.714 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2023-06-23T00:39:41.714 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-23T00:39:41.784 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T00:39:41.784 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T00:39:41.784 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T00:39:41.785 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 2023-06-23T00:39:41.785 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T00:39:41.785 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-06-23T00:39:41.785 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T00:39:41.785 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T00:39:41.785 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T00:39:41.785 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T00:39:42.242 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-06-23T00:39:42.244 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-06-23T00:39:42.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-06-23T00:39:42.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-06-23T00:39:42.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-06-23T00:39:42.470 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T00:39:42.470 INFO:tasks.ceph:role: osd.7 2023-06-23T00:39:42.470 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com 2023-06-23T00:39:42.470 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-23T00:39:42.542 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T00:39:42.543 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T00:39:42.543 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T00:39:42.543 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 2023-06-23T00:39:42.543 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T00:39:42.543 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2023-06-23T00:39:42.543 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T00:39:42.543 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T00:39:42.544 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T00:39:42.544 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T00:39:42.985 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2023-06-23T00:39:42.987 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2023-06-23T00:39:42.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-06-23T00:39:43.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-06-23T00:39:43.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:43.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:43.330+0000 7f8cd8f44540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-06-23T00:39:43.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:43.331+0000 7f8cd8f44540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-06-23T00:39:43.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:43.331+0000 7f8cd8f44540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T00:39:43.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:43.331+0000 7f8cd8f44540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T00:39:43.335 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:43.331+0000 7f8cd8f44540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T00:39:43.341 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:43.338+0000 7f8cd8f44540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-06-23T00:39:43.863 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:43.860+0000 7f8cd8f44540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:44.627+0000 7f8cd8f44540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:45.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:45.407+0000 7f8cd8f44540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:46.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-06-23T00:39:46.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:46.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:46.313+0000 7f213c8d1540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-06-23T00:39:46.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:46.313+0000 7f213c8d1540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-06-23T00:39:46.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:46.313+0000 7f213c8d1540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T00:39:46.317 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:46.313+0000 7f213c8d1540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T00:39:46.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:46.313+0000 7f213c8d1540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T00:39:46.318 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:46.314+0000 7f213c8d1540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-06-23T00:39:46.851 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:46.848+0000 7f213c8d1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:47.626 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:47.623+0000 7f213c8d1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:48.392 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:48.389+0000 7f213c8d1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:49.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-06-23T00:39:49.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:49.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:49.329+0000 7fe70d36c540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-06-23T00:39:49.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:49.329+0000 7fe70d36c540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-06-23T00:39:49.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:49.329+0000 7fe70d36c540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T00:39:49.333 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:49.329+0000 7fe70d36c540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T00:39:49.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:49.329+0000 7fe70d36c540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T00:39:49.334 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:49.330+0000 7fe70d36c540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-06-23T00:39:49.854 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:49.850+0000 7fe70d36c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:50.616 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:50.613+0000 7fe70d36c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:51.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:51.376+0000 7fe70d36c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:51.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T00:39:51.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T00:39:52.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-06-23T00:39:52.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:52.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:52.276+0000 7f7164076540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-06-23T00:39:52.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:52.276+0000 7f7164076540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-06-23T00:39:52.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:52.276+0000 7f7164076540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T00:39:52.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:52.276+0000 7f7164076540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T00:39:52.280 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:52.276+0000 7f7164076540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T00:39:52.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:52.277+0000 7f7164076540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-06-23T00:39:52.800 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:52.797+0000 7f7164076540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:53.569 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:53.565+0000 7f7164076540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:54.329+0000 7f7164076540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T00:39:55.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-06-23T00:39:55.200 INFO:tasks.ceph:Reading keys from all nodes... 2023-06-23T00:39:55.201 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:55.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-06-23T00:39:55.229 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:55.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-06-23T00:39:55.297 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:55.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-06-23T00:39:55.364 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:55.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-06-23T00:39:55.430 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:55.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-06-23T00:39:55.496 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:55.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-06-23T00:39:55.561 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:55.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-06-23T00:39:55.626 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:55.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-06-23T00:39:55.656 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:55.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-06-23T00:39:55.723 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:55.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-06-23T00:39:55.789 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:55.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-06-23T00:39:55.856 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:55.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-06-23T00:39:55.921 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:55.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-06-23T00:39:55.986 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:55.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-06-23T00:39:56.052 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:56.053 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-06-23T00:39:56.070 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:56.070 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-06-23T00:39:56.126 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:56.127 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-06-23T00:39:56.142 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:56.142 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-06-23T00:39:56.196 INFO:tasks.ceph:Adding keys to all mons... 2023-06-23T00:39:56.197 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-23T00:39:56.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-23T00:39:56.240 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.y] 2023-06-23T00:39:56.240 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC86ZRkHoyeGxAAToQFF5BAMNxVbkY6+ELfvg== 2023-06-23T00:39:56.241 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.a] 2023-06-23T00:39:56.241 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC86ZRkV7FmJRAAstfBTGTliik/FkvK1a/ToA== 2023-06-23T00:39:56.241 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.c] 2023-06-23T00:39:56.241 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC86ZRkXLZiLBAAU5x9cJfOULyuDA0BM3yTsg== 2023-06-23T00:39:56.242 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2023-06-23T00:39:56.242 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDA6ZRkDlZvGBAAd4GDCXLfqpFp7Z13Hcl6OA== 2023-06-23T00:39:56.242 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2023-06-23T00:39:56.243 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDD6ZRkmyyhFRAAtd0y1n3hi+M2hAKsu8/izg== 2023-06-23T00:39:56.243 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2023-06-23T00:39:56.243 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDG6ZRkZrtjFRAAx9PcfNMPKxElXrI0NsIMKg== 2023-06-23T00:39:56.243 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2023-06-23T00:39:56.244 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDJ6ZRkHfpYFRAAFZJxIu+QNacaTGn95cJ6xA== 2023-06-23T00:39:56.244 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2023-06-23T00:39:56.244 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC86ZRkTFo6IhAA5U3GsZ1NtgbxodRytrpl5Q== 2023-06-23T00:39:56.245 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.b] 2023-06-23T00:39:56.245 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC86ZRkqIfbMBAAU1tOdrlqbMen87KC9mYcBA== 2023-06-23T00:39:56.245 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.d] 2023-06-23T00:39:56.246 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC86ZRkbBDbNxAAHCMBPafFIK/f0McBmFaHaQ== 2023-06-23T00:39:56.246 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2023-06-23T00:39:56.246 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDP6ZRkmGzCExAAp56z24Ld1o0xui5R0Z/17Q== 2023-06-23T00:39:56.247 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2023-06-23T00:39:56.247 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDS6ZRkV/nDEhAAoYALzBIZCtyUv3z4DyL3Zg== 2023-06-23T00:39:56.247 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.6] 2023-06-23T00:39:56.247 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDV6ZRk4521ExAARy5Q3ttmoG0TRxb8Xq/vww== 2023-06-23T00:39:56.248 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.7] 2023-06-23T00:39:56.248 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDY6ZRkMeyIEBAAHfCEQYRsE797jObhVpkqOQ== 2023-06-23T00:39:56.248 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2023-06-23T00:39:56.248 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC96ZRk4vAwABAA02pPK7jUGRlWeR05WOUMKQ== 2023-06-23T00:39:56.249 INFO:teuthology.orchestra.run.smithi064.stdout:[client.2] 2023-06-23T00:39:56.249 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC96ZRk1KesBRAAqmQ8H/cE72vfHnibc1/FXw== 2023-06-23T00:39:56.249 INFO:teuthology.orchestra.run.smithi064.stdout:[client.1] 2023-06-23T00:39:56.249 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC96ZRk9QVHChAA4tYYY0tfsrWPdE93LBAR5g== 2023-06-23T00:39:56.250 INFO:teuthology.orchestra.run.smithi064.stdout:[client.3] 2023-06-23T00:39:56.250 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC96ZRk0eG6DxAAPjHBSnajCkoc5RpiwC6XJQ== 2023-06-23T00:39:56.259 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.y] 2023-06-23T00:39:56.259 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC86ZRkHoyeGxAAToQFF5BAMNxVbkY6+ELfvg== 2023-06-23T00:39:56.259 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.a] 2023-06-23T00:39:56.259 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC86ZRkV7FmJRAAstfBTGTliik/FkvK1a/ToA== 2023-06-23T00:39:56.260 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.c] 2023-06-23T00:39:56.260 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC86ZRkXLZiLBAAU5x9cJfOULyuDA0BM3yTsg== 2023-06-23T00:39:56.260 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2023-06-23T00:39:56.260 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDA6ZRkDlZvGBAAd4GDCXLfqpFp7Z13Hcl6OA== 2023-06-23T00:39:56.260 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2023-06-23T00:39:56.260 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDD6ZRkmyyhFRAAtd0y1n3hi+M2hAKsu8/izg== 2023-06-23T00:39:56.261 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2023-06-23T00:39:56.261 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDG6ZRkZrtjFRAAx9PcfNMPKxElXrI0NsIMKg== 2023-06-23T00:39:56.261 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.3] 2023-06-23T00:39:56.261 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDJ6ZRkHfpYFRAAFZJxIu+QNacaTGn95cJ6xA== 2023-06-23T00:39:56.261 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2023-06-23T00:39:56.261 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC86ZRkTFo6IhAA5U3GsZ1NtgbxodRytrpl5Q== 2023-06-23T00:39:56.261 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.b] 2023-06-23T00:39:56.262 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC86ZRkqIfbMBAAU1tOdrlqbMen87KC9mYcBA== 2023-06-23T00:39:56.262 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.d] 2023-06-23T00:39:56.262 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC86ZRkbBDbNxAAHCMBPafFIK/f0McBmFaHaQ== 2023-06-23T00:39:56.262 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.4] 2023-06-23T00:39:56.262 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDP6ZRkmGzCExAAp56z24Ld1o0xui5R0Z/17Q== 2023-06-23T00:39:56.262 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.5] 2023-06-23T00:39:56.262 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDS6ZRkV/nDEhAAoYALzBIZCtyUv3z4DyL3Zg== 2023-06-23T00:39:56.263 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.6] 2023-06-23T00:39:56.263 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDV6ZRk4521ExAARy5Q3ttmoG0TRxb8Xq/vww== 2023-06-23T00:39:56.263 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.7] 2023-06-23T00:39:56.263 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDY6ZRkMeyIEBAAHfCEQYRsE797jObhVpkqOQ== 2023-06-23T00:39:56.263 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2023-06-23T00:39:56.263 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC96ZRk4vAwABAA02pPK7jUGRlWeR05WOUMKQ== 2023-06-23T00:39:56.263 INFO:teuthology.orchestra.run.smithi086.stdout:[client.2] 2023-06-23T00:39:56.264 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC96ZRk1KesBRAAqmQ8H/cE72vfHnibc1/FXw== 2023-06-23T00:39:56.264 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2023-06-23T00:39:56.264 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC96ZRk9QVHChAA4tYYY0tfsrWPdE93LBAR5g== 2023-06-23T00:39:56.264 INFO:teuthology.orchestra.run.smithi086.stdout:[client.3] 2023-06-23T00:39:56.264 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC96ZRk0eG6DxAAPjHBSnajCkoc5RpiwC6XJQ== 2023-06-23T00:39:56.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T00:39:56.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T00:39:56.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T00:39:56.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T00:39:56.406 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T00:39:56.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T00:39:56.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T00:39:56.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T00:39:56.708 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T00:39:56.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T00:39:56.760 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T00:39:56.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T00:39:56.812 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.861 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.962 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:56.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T00:39:57.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T00:39:57.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T00:39:57.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T00:39:57.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T00:39:57.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T00:39:57.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T00:39:57.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T00:39:57.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T00:39:57.221 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-06-23T00:39:57.221 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-06-23T00:39:57.246 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T00:39:57.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-06-23T00:39:57.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-06-23T00:39:57.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T00:39:57.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-06-23T00:39:57.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-06-23T00:39:57.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T00:39:57.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-06-23T00:39:57.686 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:57.688 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-23T00:39:57.740 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-06-23T00:39:57.741 INFO:tasks.ceph.mon.a:Restarting daemon 2023-06-23T00:39:57.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-06-23T00:39:57.744 INFO:tasks.ceph.mon.a:Started 2023-06-23T00:39:57.745 INFO:tasks.ceph.mon.b:Restarting daemon 2023-06-23T00:39:57.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-06-23T00:39:57.782 INFO:tasks.ceph.mon.b:Started 2023-06-23T00:39:57.783 INFO:tasks.ceph.mon.c:Restarting daemon 2023-06-23T00:39:57.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-06-23T00:39:57.784 INFO:tasks.ceph.mon.c:Started 2023-06-23T00:39:57.785 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-06-23T00:39:57.785 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-06-23T00:39:57.786 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-06-23T00:39:57.788 INFO:tasks.ceph.mgr.y:Started 2023-06-23T00:39:57.789 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-06-23T00:39:57.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-06-23T00:39:57.791 INFO:tasks.ceph.mgr.x:Started 2023-06-23T00:39:57.792 INFO:tasks.ceph:Setting crush tunables to default 2023-06-23T00:39:57.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd crush tunables default 2023-06-23T00:39:57.912 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.908+0000 7f64df59e700 1 Processor -- start 2023-06-23T00:39:57.912 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.908+0000 7f64df59e700 1 -- start start 2023-06-23T00:39:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.909+0000 7f64df59e700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f64e0155530 0x7f64e00c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.909+0000 7f64df59e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 0x7f64e006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.909+0000 7f64df59e700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e006ec10 con 0x7f64e006e7b0 2023-06-23T00:39:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.909+0000 7f64df59e700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e00aff60 con 0x7f64e0155530 2023-06-23T00:39:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.909+0000 7f64df59e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00accb0 con 0x7f64e0155900 2023-06-23T00:39:57.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.909+0000 7f64de59c700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f64e0155530 0x7f64e00c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.909+0000 7f64ded9d700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f64e006e7b0 0x7f64e0069e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:52496/0 (socket says 172.21.15.64:52496) 2023-06-23T00:39:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.909+0000 7f64ded9d700 1 -- 172.21.15.64:0/1055804246 learned_addr learned my addr 172.21.15.64:0/1055804246 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:39:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.910+0000 7f64de59c700 1 -- 172.21.15.64:0/1055804246 >> v2:172.21.15.64:3300/0 conn(0x7f64e0155530 msgr2=0x7f64e00c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-23T00:39:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.910+0000 7f64de59c700 1 -- 172.21.15.64:0/1055804246 >> v2:172.21.15.64:3300/0 conn(0x7f64e0155530 msgr2=0x7f64e00c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T00:39:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.910+0000 7f64de59c700 1 --2- 172.21.15.64:0/1055804246 >> v2:172.21.15.64:3300/0 conn(0x7f64e0155530 0x7f64e00c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T00:39:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.910+0000 7f64de59c700 1 --2- 172.21.15.64:0/1055804246 >> v2:172.21.15.64:3300/0 conn(0x7f64e0155530 0x7f64e00c8a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T00:39:57.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.910+0000 7f64ddd9b700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 0x7f64e006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.910+0000 7f64ddd9b700 1 -- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 msgr2=0x7f64e006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-06-23T00:39:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.910+0000 7f64ddd9b700 1 -- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 msgr2=0x7f64e006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T00:39:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.910+0000 7f64ddd9b700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 0x7f64e006e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T00:39:57.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:57.910+0000 7f64ddd9b700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 0x7f64e006e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T00:39:58.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.102+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825155727 0 0) 0x7f64d8001490 con 0x7f64e006e7b0 2023-06-23T00:39:58.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.102+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64c8003570 con 0x7f64e006e7b0 2023-06-23T00:39:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1735197630 0 0) 0x7f64c8003570 con 0x7f64e006e7b0 2023-06-23T00:39:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 msgr2=0x7f64e006e270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-23T00:39:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 0x7f64e006e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 >> v2:172.21.15.64:3300/0 conn(0x7f64e0155530 msgr2=0x7f64e00c8a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-23T00:39:58.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 --2- 172.21.15.64:0/1055804246 >> v2:172.21.15.64:3300/0 conn(0x7f64e0155530 0x7f64e00c8a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e00af350 con 0x7f64e006e7b0 2023-06-23T00:39:58.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 939283155 0 0) 0x7f64d8001980 con 0x7f64e006e7b0 2023-06-23T00:39:58.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 >> v1:172.21.15.64:6789/0 conn(0x7f64e006e7b0 legacy=0x7f64e0069e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:58.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64c8003a60 0x7f64c8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:58.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 0x7f64c8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:58.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c8004d50 0x7f64c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:58.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00aff60 con 0x7f64c8004d50 2023-06-23T00:39:58.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00accb0 con 0x7f64c8003a60 2023-06-23T00:39:58.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.104+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c800bb90 con 0x7f64e0155900 2023-06-23T00:39:58.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64ded9d700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c8004d50 0x7f64c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:58.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64ddd9b700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 0x7f64c8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:58.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64de59c700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64c8003a60 0x7f64c8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:58.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64ddd9b700 1 -- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64c8003a60 msgr2=0x7f64c8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:39:58.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64ddd9b700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64c8003a60 0x7f64c8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64ddd9b700 1 -- 172.21.15.64:0/1055804246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c8004d50 msgr2=0x7f64c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:39:58.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64ddd9b700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c8004d50 0x7f64c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64ddd9b700 1 -- 172.21.15.64:0/1055804246 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c8003570 con 0x7f64e0155900 2023-06-23T00:39:58.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64ded9d700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64c8004d50 0x7f64c80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:39:58.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64ddd9b700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 0x7f64c8004810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f64d000bb70 tx=0x7f64d0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=392d8aa5d51dc3ef server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:58.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d0018070 con 0x7f64e0155900 2023-06-23T00:39:58.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d000d5d0 con 0x7f64e0155900 2023-06-23T00:39:58.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.105+0000 7f64dd59a700 1 -- 172.21.15.64:0/1055804246 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d000d970 con 0x7f64e0155900 2023-06-23T00:39:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.106+0000 7f64df59e700 1 -- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 msgr2=0x7f64c8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.106+0000 7f64df59e700 1 --2- 172.21.15.64:0/1055804246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e0155900 0x7f64c8004810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f64d000bb70 tx=0x7f64d0008040 comp rx=0 tx=0).stop 2023-06-23T00:39:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.106+0000 7f64ded9d700 1 -- 172.21.15.64:0/1055804246 reap_dead start 2023-06-23T00:39:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.106+0000 7f64df59e700 1 -- 172.21.15.64:0/1055804246 shutdown_connections 2023-06-23T00:39:58.115 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.106+0000 7f64df59e700 1 -- 172.21.15.64:0/1055804246 >> 172.21.15.64:0/1055804246 conn(0x7f64e0063020 msgr2=0x7f64e0063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.106+0000 7f64df59e700 1 -- 172.21.15.64:0/1055804246 shutdown_connections 2023-06-23T00:39:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.106+0000 7f64df59e700 1 -- 172.21.15.64:0/1055804246 wait complete. 2023-06-23T00:39:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.108+0000 7f64df59e700 1 Processor -- start 2023-06-23T00:39:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.108+0000 7f64df59e700 1 -- start start 2023-06-23T00:39:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64df59e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64e0155900 0x7f64e00ab520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64df59e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64e015b5e0 0x7f64e0169470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64df59e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e01699b0 0x7f64e01515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:58.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64df59e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e016a410 con 0x7f64e015b5e0 2023-06-23T00:39:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64df59e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e015b150 con 0x7f64e0155900 2023-06-23T00:39:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64df59e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e016a130 con 0x7f64e01699b0 2023-06-23T00:39:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64ddd9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64e015b5e0 0x7f64e0169470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64ddd9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64e015b5e0 0x7f64e0169470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:36074/0 (socket says 172.21.15.64:36074) 2023-06-23T00:39:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64ddd9b700 1 -- 172.21.15.64:0/396161063 learned_addr learned my addr 172.21.15.64:0/396161063 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:39:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64ded9d700 1 --2- 172.21.15.64:0/396161063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e01699b0 0x7f64e01515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:58.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64de59c700 1 --2- 172.21.15.64:0/396161063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64e0155900 0x7f64e00ab520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64ddd9b700 1 -- 172.21.15.64:0/396161063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e01699b0 msgr2=0x7f64e01515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64ddd9b700 1 --2- 172.21.15.64:0/396161063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e01699b0 0x7f64e01515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64ddd9b700 1 -- 172.21.15.64:0/396161063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64e0155900 msgr2=0x7f64e00ab520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64ddd9b700 1 --2- 172.21.15.64:0/396161063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64e0155900 0x7f64e00ab520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.109+0000 7f64ddd9b700 1 -- 172.21.15.64:0/396161063 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e015ab00 con 0x7f64e015b5e0 2023-06-23T00:39:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.110+0000 7f64ded9d700 1 --2- 172.21.15.64:0/396161063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e01699b0 0x7f64e01515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:39:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.110+0000 7f64ddd9b700 1 --2- 172.21.15.64:0/396161063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64e015b5e0 0x7f64e0169470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f64d000c010 tx=0x7f64d001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:58.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.110+0000 7f64ce7fc700 1 -- 172.21.15.64:0/396161063 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d001f500 con 0x7f64e015b5e0 2023-06-23T00:39:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.110+0000 7f64ce7fc700 1 -- 172.21.15.64:0/396161063 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d0023070 con 0x7f64e015b5e0 2023-06-23T00:39:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.110+0000 7f64df59e700 1 -- 172.21.15.64:0/396161063 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e015ade0 con 0x7f64e015b5e0 2023-06-23T00:39:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.110+0000 7f64ce7fc700 1 -- 172.21.15.64:0/396161063 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d001f8b0 con 0x7f64e015b5e0 2023-06-23T00:39:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.110+0000 7f64df59e700 1 -- 172.21.15.64:0/396161063 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e00aba60 con 0x7f64e015b5e0 2023-06-23T00:39:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.110+0000 7f64ce7fc700 1 -- 172.21.15.64:0/396161063 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f64d0025030 con 0x7f64e015b5e0 2023-06-23T00:39:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.110+0000 7f64ce7fc700 1 -- 172.21.15.64:0/396161063 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f64d00287d0 con 0x7f64e015b5e0 2023-06-23T00:39:58.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.111+0000 7f64df59e700 1 -- 172.21.15.64:0/396161063 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e015a960 con 0x7f64e015b5e0 2023-06-23T00:39:58.120 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.113+0000 7f64ce7fc700 1 -- 172.21.15.64:0/396161063 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f64e015a960 con 0x7f64e015b5e0 2023-06-23T00:39:58.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.202+0000 7f64df59e700 1 -- 172.21.15.64:0/396161063 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f64e00abc50 con 0x7f64e015b5e0 2023-06-23T00:39:58.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.209+0000 7f64ce7fc700 1 -- 172.21.15.64:0/396161063 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f64d002d020 con 0x7f64e015b5e0 2023-06-23T00:39:58.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.210+0000 7f64df59e700 1 -- 172.21.15.64:0/396161063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64e015b5e0 msgr2=0x7f64e0169470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:58.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.210+0000 7f64df59e700 1 --2- 172.21.15.64:0/396161063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64e015b5e0 0x7f64e0169470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f64d000c010 tx=0x7f64d001a000 comp rx=0 tx=0).stop 2023-06-23T00:39:58.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.210+0000 7f64df59e700 1 -- 172.21.15.64:0/396161063 shutdown_connections 2023-06-23T00:39:58.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.210+0000 7f64df59e700 1 --2- 172.21.15.64:0/396161063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64e01699b0 0x7f64e01515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.210+0000 7f64df59e700 1 --2- 172.21.15.64:0/396161063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64e0155900 0x7f64e00ab520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.210+0000 7f64df59e700 1 --2- 172.21.15.64:0/396161063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64e015b5e0 0x7f64e0169470 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.210+0000 7f64df59e700 1 -- 172.21.15.64:0/396161063 >> 172.21.15.64:0/396161063 conn(0x7f64e0063020 msgr2=0x7f64e0152530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:58.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.210+0000 7f64df59e700 1 -- 172.21.15.64:0/396161063 shutdown_connections 2023-06-23T00:39:58.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:39:58.210+0000 7f64df59e700 1 -- 172.21.15.64:0/396161063 wait complete. 2023-06-23T00:39:58.215 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2023-06-23T00:39:58.225 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-06-23T00:39:58.225 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:58.226 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-06-23T00:39:58.252 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:58.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-06-23T00:39:58.317 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:58.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-06-23T00:39:58.382 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-06-23T00:39:58.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-06-23T00:39:58.447 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:58.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-06-23T00:39:58.472 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:58.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-06-23T00:39:58.537 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:58.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-06-23T00:39:58.602 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2023-06-23T00:39:58.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-06-23T00:39:58.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37 0 2023-06-23T00:39:58.823 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.820+0000 7f5b4db8f700 1 Processor -- start 2023-06-23T00:39:58.823 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.820+0000 7f5b4db8f700 1 -- start start 2023-06-23T00:39:58.825 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b4db8f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b481570c0 0x7f5b4815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:58.825 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b4db8f700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4815aaa0 con 0x7f5b480d04f0 2023-06-23T00:39:58.825 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b4db8f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b480ce5c0 con 0x7f5b480d08c0 2023-06-23T00:39:58.825 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b4db8f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b480cd1d0 con 0x7f5b481570c0 2023-06-23T00:39:58.825 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b47fff700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5b480d08c0 0x7f5b480cea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40146/0 (socket says 172.21.15.86:40146) 2023-06-23T00:39:58.826 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b46ffd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b481570c0 0x7f5b4815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:58.826 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b47fff700 1 -- 172.21.15.86:0/633955693 learned_addr learned my addr 172.21.15.86:0/633955693 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-06-23T00:39:58.826 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b467fc700 1 -- 172.21.15.86:0/633955693 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640796443 0 0) 0x7f5b480ce5c0 con 0x7f5b480d08c0 2023-06-23T00:39:58.826 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b467fc700 1 -- 172.21.15.86:0/633955693 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b30003570 con 0x7f5b480d08c0 2023-06-23T00:39:58.826 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b46ffd700 1 -- 172.21.15.86:0/633955693 >> v1:172.21.15.86:6789/0 conn(0x7f5b480d08c0 legacy=0x7f5b480cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:58.826 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b46ffd700 1 -- 172.21.15.86:0/633955693 >> v1:172.21.15.64:6789/0 conn(0x7f5b480d04f0 legacy=0x7f5b481569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:58.826 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b46ffd700 1 -- 172.21.15.86:0/633955693 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b480cd4b0 con 0x7f5b481570c0 2023-06-23T00:39:58.827 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b46ffd700 1 --2- 172.21.15.86:0/633955693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b481570c0 0x7f5b4815a410 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5b38009910 tx=0x7f5b380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c8acface8663fff server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:58.827 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b467fc700 1 -- 172.21.15.86:0/633955693 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38011630 con 0x7f5b481570c0 2023-06-23T00:39:58.827 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b467fc700 1 -- 172.21.15.86:0/633955693 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b38011790 con 0x7f5b481570c0 2023-06-23T00:39:58.827 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.821+0000 7f5b467fc700 1 -- 172.21.15.86:0/633955693 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38011b50 con 0x7f5b481570c0 2023-06-23T00:39:58.831 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.822+0000 7f5b4db8f700 1 -- 172.21.15.86:0/633955693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b481570c0 msgr2=0x7f5b4815a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:58.831 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.822+0000 7f5b4db8f700 1 --2- 172.21.15.86:0/633955693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b481570c0 0x7f5b4815a410 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5b38009910 tx=0x7f5b380091a0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.832 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.822+0000 7f5b4db8f700 1 -- 172.21.15.86:0/633955693 shutdown_connections 2023-06-23T00:39:58.832 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.822+0000 7f5b4db8f700 1 --2- 172.21.15.86:0/633955693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b481570c0 0x7f5b4815a410 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.832 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.822+0000 7f5b4db8f700 1 -- 172.21.15.86:0/633955693 >> 172.21.15.86:0/633955693 conn(0x7f5b4806a060 msgr2=0x7f5b4806a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:58.833 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.822+0000 7f5b4db8f700 1 -- 172.21.15.86:0/633955693 shutdown_connections 2023-06-23T00:39:58.833 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.822+0000 7f5b4db8f700 1 -- 172.21.15.86:0/633955693 wait complete. 2023-06-23T00:39:58.833 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.823+0000 7f5b4db8f700 1 Processor -- start 2023-06-23T00:39:58.834 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.823+0000 7f5b4db8f700 1 -- start start 2023-06-23T00:39:58.834 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b4db8f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b480d04f0 0x7f5b48147cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:58.834 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b4db8f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b480d08c0 0x7f5b480b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:58.834 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b4db8f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b481570c0 0x7f5b480b3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:58.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b4db8f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b480b5ef0 con 0x7f5b481570c0 2023-06-23T00:39:58.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b4db8f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4816b360 con 0x7f5b480d04f0 2023-06-23T00:39:58.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b4db8f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4816a790 con 0x7f5b480d08c0 2023-06-23T00:39:58.835 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b46ffd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b480d08c0 0x7f5b480b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:58.836 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b477fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b480d04f0 0x7f5b48147cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:58.836 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b46ffd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b480d08c0 0x7f5b480b3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48318/0 (socket says 172.21.15.86:48318) 2023-06-23T00:39:58.836 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b46ffd700 1 -- 172.21.15.86:0/189335177 learned_addr learned my addr 172.21.15.86:0/189335177 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-06-23T00:39:58.836 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b477fe700 1 --2- 172.21.15.86:0/189335177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b480d04f0 0x7f5b48147cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38518/0 (socket says 172.21.15.86:38518) 2023-06-23T00:39:58.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.824+0000 7f5b47fff700 1 --2- 172.21.15.86:0/189335177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b481570c0 0x7f5b480b3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:58.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b477fe700 1 -- 172.21.15.86:0/189335177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b480d08c0 msgr2=0x7f5b480b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:58.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b477fe700 1 --2- 172.21.15.86:0/189335177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b480d08c0 0x7f5b480b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.837 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b477fe700 1 -- 172.21.15.86:0/189335177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b481570c0 msgr2=0x7f5b480b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:58.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b477fe700 1 --2- 172.21.15.86:0/189335177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b481570c0 0x7f5b480b3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b477fe700 1 -- 172.21.15.86:0/189335177 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b38009420 con 0x7f5b480d04f0 2023-06-23T00:39:58.838 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b46ffd700 1 --2- 172.21.15.86:0/189335177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b480d08c0 0x7f5b480b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:39:58.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b477fe700 1 --2- 172.21.15.86:0/189335177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b480d04f0 0x7f5b48147cb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00c7e0 tx=0x7f5b3c007ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:58.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b2f7fe700 1 -- 172.21.15.86:0/189335177 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c011590 con 0x7f5b480d04f0 2023-06-23T00:39:58.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b2f7fe700 1 -- 172.21.15.86:0/189335177 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3c011730 con 0x7f5b480d04f0 2023-06-23T00:39:58.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b4db8f700 1 -- 172.21.15.86:0/189335177 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4816aa30 con 0x7f5b480d04f0 2023-06-23T00:39:58.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b2f7fe700 1 -- 172.21.15.86:0/189335177 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c011af0 con 0x7f5b480d04f0 2023-06-23T00:39:58.839 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b4db8f700 1 -- 172.21.15.86:0/189335177 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4816aff0 con 0x7f5b480d04f0 2023-06-23T00:39:58.840 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b2f7fe700 1 -- 172.21.15.86:0/189335177 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5b3c01c6c0 con 0x7f5b480d04f0 2023-06-23T00:39:58.840 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.825+0000 7f5b2f7fe700 1 -- 172.21.15.86:0/189335177 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5b3c012770 con 0x7f5b480d04f0 2023-06-23T00:39:58.840 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.826+0000 7f5b4db8f700 1 -- 172.21.15.86:0/189335177 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4816b640 con 0x7f5b480d04f0 2023-06-23T00:39:58.841 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.828+0000 7f5b2f7fe700 1 -- 172.21.15.86:0/189335177 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f5b3c021060 con 0x7f5b480d04f0 2023-06-23T00:39:58.914 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.911+0000 7f5b4db8f700 1 -- 172.21.15.86:0/189335177 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37", "id": 0} v 0) v1 -- 0x7f5b4804c1b0 con 0x7f5b480d04f0 2023-06-23T00:39:58.920 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.917+0000 7f5b2f7fe700 1 -- 172.21.15.86:0/189335177 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5b3c02a0b0 con 0x7f5b480d04f0 2023-06-23T00:39:58.921 INFO:teuthology.orchestra.run.smithi086.stdout:0 2023-06-23T00:39:58.922 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.917+0000 7f5b4db8f700 1 -- 172.21.15.86:0/189335177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b480d04f0 msgr2=0x7f5b48147cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:58.922 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.917+0000 7f5b4db8f700 1 --2- 172.21.15.86:0/189335177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b480d04f0 0x7f5b48147cb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00c7e0 tx=0x7f5b3c007ea0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.922 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.917+0000 7f5b4db8f700 1 -- 172.21.15.86:0/189335177 shutdown_connections 2023-06-23T00:39:58.923 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.917+0000 7f5b4db8f700 1 --2- 172.21.15.86:0/189335177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b481570c0 0x7f5b480b3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.923 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.917+0000 7f5b4db8f700 1 --2- 172.21.15.86:0/189335177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b480d04f0 0x7f5b48147cb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.923 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.917+0000 7f5b4db8f700 1 --2- 172.21.15.86:0/189335177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b480d08c0 0x7f5b480b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:58.924 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.917+0000 7f5b4db8f700 1 -- 172.21.15.86:0/189335177 >> 172.21.15.86:0/189335177 conn(0x7f5b4806a060 msgr2=0x7f5b4815acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:58.927 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.918+0000 7f5b4db8f700 1 -- 172.21.15.86:0/189335177 shutdown_connections 2023-06-23T00:39:58.928 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:58.918+0000 7f5b4db8f700 1 -- 172.21.15.86:0/189335177 wait complete. 2023-06-23T00:39:58.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new f0698c9d-c67a-42ac-8d36-838836c514d6 1 2023-06-23T00:39:59.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.044+0000 7f22e4b12700 1 Processor -- start 2023-06-23T00:39:59.048 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.044+0000 7f22e4b12700 1 -- start start 2023-06-23T00:39:59.049 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.045+0000 7f22e4b12700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f22e01590c0 0x7f22e0159490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.045+0000 7f22e4b12700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f22e006d050 0x7f22e0068980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.045+0000 7f22e4b12700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e0068ec0 con 0x7f22e0068510 2023-06-23T00:39:59.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.045+0000 7f22e4b12700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e015dd90 con 0x7f22e006d050 2023-06-23T00:39:59.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.045+0000 7f22e4b12700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e015d110 con 0x7f22e01590c0 2023-06-23T00:39:59.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.045+0000 7f22de59c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f22e01590c0 0x7f22e0159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22de59c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f22e01590c0 0x7f22e0159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38530/0 (socket says 172.21.15.86:38530) 2023-06-23T00:39:59.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22ddd9b700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f22e0068510 0x7f22e006c940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40158/0 (socket says 172.21.15.86:40158) 2023-06-23T00:39:59.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22de59c700 1 -- 172.21.15.86:0/2155077588 learned_addr learned my addr 172.21.15.86:0/2155077588 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-06-23T00:39:59.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22ded9d700 1 --2- 172.21.15.86:0/2155077588 >> v2:172.21.15.64:3300/0 conn(0x7f22e006d050 0x7f22e0068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22de59c700 1 -- 172.21.15.86:0/2155077588 >> v2:172.21.15.64:3300/0 conn(0x7f22e006d050 msgr2=0x7f22e0068980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:39:59.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22de59c700 1 --2- 172.21.15.86:0/2155077588 >> v2:172.21.15.64:3300/0 conn(0x7f22e006d050 0x7f22e0068980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22de59c700 1 -- 172.21.15.86:0/2155077588 >> v1:172.21.15.86:6789/0 conn(0x7f22e0068510 legacy=0x7f22e006c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22de59c700 1 -- 172.21.15.86:0/2155077588 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e015d3f0 con 0x7f22e01590c0 2023-06-23T00:39:59.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22dd59a700 1 -- 172.21.15.86:0/2155077588 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270236435 0 0) 0x7f22e0068ec0 con 0x7f22e0068510 2023-06-23T00:39:59.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22de59c700 1 --2- 172.21.15.86:0/2155077588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01590c0 0x7f22e0159490 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f22e00af9d0 tx=0x7f22c8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb2a61d9ca56aac9 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:59.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22dd59a700 1 -- 172.21.15.86:0/2155077588 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c8009e70 con 0x7f22e01590c0 2023-06-23T00:39:59.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22dd59a700 1 -- 172.21.15.86:0/2155077588 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c800b860 con 0x7f22e01590c0 2023-06-23T00:39:59.052 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.046+0000 7f22dd59a700 1 -- 172.21.15.86:0/2155077588 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c800bc00 con 0x7f22e01590c0 2023-06-23T00:39:59.057 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.047+0000 7f22e4b12700 1 -- 172.21.15.86:0/2155077588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01590c0 msgr2=0x7f22e0159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.058 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.047+0000 7f22e4b12700 1 --2- 172.21.15.86:0/2155077588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01590c0 0x7f22e0159490 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f22e00af9d0 tx=0x7f22c8009250 comp rx=0 tx=0).stop 2023-06-23T00:39:59.058 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.047+0000 7f22e4b12700 1 -- 172.21.15.86:0/2155077588 shutdown_connections 2023-06-23T00:39:59.058 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.047+0000 7f22e4b12700 1 --2- 172.21.15.86:0/2155077588 >> v2:172.21.15.64:3300/0 conn(0x7f22e006d050 0x7f22e0068980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.059 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.047+0000 7f22e4b12700 1 --2- 172.21.15.86:0/2155077588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e01590c0 0x7f22e0159490 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.059 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.047+0000 7f22e4b12700 1 -- 172.21.15.86:0/2155077588 >> 172.21.15.86:0/2155077588 conn(0x7f22e006a0c0 msgr2=0x7f22e006b100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:59.059 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.047+0000 7f22e4b12700 1 -- 172.21.15.86:0/2155077588 shutdown_connections 2023-06-23T00:39:59.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.047+0000 7f22e4b12700 1 -- 172.21.15.86:0/2155077588 wait complete. 2023-06-23T00:39:59.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.048+0000 7f22e4b12700 1 Processor -- start 2023-06-23T00:39:59.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.048+0000 7f22e4b12700 1 -- start start 2023-06-23T00:39:59.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22e4b12700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e0068510 0x7f22e0158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22e4b12700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22e006d050 0x7f22e0155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.060 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22e4b12700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22e01590c0 0x7f22e0140120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22e4b12700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e01567a0 con 0x7f22e006d050 2023-06-23T00:39:59.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22e4b12700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e0156a80 con 0x7f22e0068510 2023-06-23T00:39:59.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22e4b12700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e0156d60 con 0x7f22e01590c0 2023-06-23T00:39:59.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22de59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e0068510 0x7f22e0158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22de59c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e0068510 0x7f22e0158150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38532/0 (socket says 172.21.15.86:38532) 2023-06-23T00:39:59.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22de59c700 1 -- 172.21.15.86:0/1937259033 learned_addr learned my addr 172.21.15.86:0/1937259033 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-06-23T00:39:59.061 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22ded9d700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22e01590c0 0x7f22e0140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22ddd9b700 1 --2- 172.21.15.86:0/1937259033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22e006d050 0x7f22e0155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22de59c700 1 -- 172.21.15.86:0/1937259033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22e01590c0 msgr2=0x7f22e0140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22de59c700 1 --2- 172.21.15.86:0/1937259033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22e01590c0 0x7f22e0140120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22de59c700 1 -- 172.21.15.86:0/1937259033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22e006d050 msgr2=0x7f22e0155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22de59c700 1 --2- 172.21.15.86:0/1937259033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22e006d050 0x7f22e0155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22de59c700 1 -- 172.21.15.86:0/1937259033 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c80094f0 con 0x7f22e0068510 2023-06-23T00:39:59.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22ded9d700 1 --2- 172.21.15.86:0/1937259033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22e01590c0 0x7f22e0140120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:39:59.062 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22ddd9b700 1 --2- 172.21.15.86:0/1937259033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22e006d050 0x7f22e0155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:39:59.063 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22de59c700 1 --2- 172.21.15.86:0/1937259033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e0068510 0x7f22e0158150 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f22c80094c0 tx=0x7f22c8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:59.063 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22d67fc700 1 -- 172.21.15.86:0/1937259033 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c800bbe0 con 0x7f22e0068510 2023-06-23T00:39:59.063 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22d67fc700 1 -- 172.21.15.86:0/1937259033 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c8000c90 con 0x7f22e0068510 2023-06-23T00:39:59.063 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22e4b12700 1 -- 172.21.15.86:0/1937259033 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e016aa80 con 0x7f22e0068510 2023-06-23T00:39:59.063 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22d67fc700 1 -- 172.21.15.86:0/1937259033 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c8011650 con 0x7f22e0068510 2023-06-23T00:39:59.063 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.049+0000 7f22e4b12700 1 -- 172.21.15.86:0/1937259033 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e0156440 con 0x7f22e0068510 2023-06-23T00:39:59.064 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.050+0000 7f22d67fc700 1 -- 172.21.15.86:0/1937259033 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f22c8011cc0 con 0x7f22e0068510 2023-06-23T00:39:59.064 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.050+0000 7f22d67fc700 1 -- 172.21.15.86:0/1937259033 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f22c802ec10 con 0x7f22e0068510 2023-06-23T00:39:59.064 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.050+0000 7f22e4b12700 1 -- 172.21.15.86:0/1937259033 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e016af20 con 0x7f22e0068510 2023-06-23T00:39:59.064 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.052+0000 7f22d67fc700 1 -- 172.21.15.86:0/1937259033 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f22c8033060 con 0x7f22e0068510 2023-06-23T00:39:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.136+0000 7f22e4b12700 1 -- 172.21.15.86:0/1937259033 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f0698c9d-c67a-42ac-8d36-838836c514d6", "id": 1} v 0) v1 -- 0x7f22e004c1b0 con 0x7f22e0068510 2023-06-23T00:39:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.139+0000 7f22d67fc700 1 -- 172.21.15.86:0/1937259033 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f0698c9d-c67a-42ac-8d36-838836c514d6", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f22c8015020 con 0x7f22e0068510 2023-06-23T00:39:59.144 INFO:teuthology.orchestra.run.smithi086.stdout:1 2023-06-23T00:39:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.140+0000 7f22e4b12700 1 -- 172.21.15.86:0/1937259033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e0068510 msgr2=0x7f22e0158150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.140+0000 7f22e4b12700 1 --2- 172.21.15.86:0/1937259033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e0068510 0x7f22e0158150 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f22c80094c0 tx=0x7f22c8017000 comp rx=0 tx=0).stop 2023-06-23T00:39:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.140+0000 7f22e4b12700 1 -- 172.21.15.86:0/1937259033 shutdown_connections 2023-06-23T00:39:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.140+0000 7f22e4b12700 1 --2- 172.21.15.86:0/1937259033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22e01590c0 0x7f22e0140120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.140+0000 7f22e4b12700 1 --2- 172.21.15.86:0/1937259033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22e0068510 0x7f22e0158150 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.140+0000 7f22e4b12700 1 --2- 172.21.15.86:0/1937259033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22e006d050 0x7f22e0155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.140+0000 7f22e4b12700 1 -- 172.21.15.86:0/1937259033 >> 172.21.15.86:0/1937259033 conn(0x7f22e006a0c0 msgr2=0x7f22e00aec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.140+0000 7f22e4b12700 1 -- 172.21.15.86:0/1937259033 shutdown_connections 2023-06-23T00:39:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.140+0000 7f22e4b12700 1 -- 172.21.15.86:0/1937259033 wait complete. 2023-06-23T00:39:59.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new a5ece869-9da7-40e8-8f1d-bbaf67bb2187 2 2023-06-23T00:39:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.270+0000 7fe809312700 1 Processor -- start 2023-06-23T00:39:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.270+0000 7fe809312700 1 -- start start 2023-06-23T00:39:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe809312700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fe8040d08c0 0x7fe80415a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe809312700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80415aaa0 con 0x7fe8040d04f0 2023-06-23T00:39:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe809312700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8040ce5c0 con 0x7fe8041570c0 2023-06-23T00:39:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe809312700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8040cd1d0 con 0x7fe8040d08c0 2023-06-23T00:39:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe8027fc700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe8041570c0 0x7fe8040cea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40168/0 (socket says 172.21.15.86:40168) 2023-06-23T00:39:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe8027fc700 1 -- 172.21.15.86:0/116890432 learned_addr learned my addr 172.21.15.86:0/116890432 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-06-23T00:39:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe8037fe700 1 --2- 172.21.15.86:0/116890432 >> v2:172.21.15.64:3300/0 conn(0x7fe8040d08c0 0x7fe80415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe801ffb700 1 -- 172.21.15.86:0/116890432 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352581299 0 0) 0x7fe8040ce5c0 con 0x7fe8041570c0 2023-06-23T00:39:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe801ffb700 1 -- 172.21.15.86:0/116890432 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7ec003570 con 0x7fe8041570c0 2023-06-23T00:39:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe801ffb700 1 -- 172.21.15.86:0/116890432 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195592707 0 0) 0x7fe80415aaa0 con 0x7fe8040d04f0 2023-06-23T00:39:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe801ffb700 1 -- 172.21.15.86:0/116890432 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8040ce5c0 con 0x7fe8040d04f0 2023-06-23T00:39:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe8037fe700 1 -- 172.21.15.86:0/116890432 >> v1:172.21.15.86:6789/0 conn(0x7fe8041570c0 legacy=0x7fe8040cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.271+0000 7fe8037fe700 1 -- 172.21.15.86:0/116890432 >> v1:172.21.15.64:6789/0 conn(0x7fe8040d04f0 legacy=0x7fe8041569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe8037fe700 1 -- 172.21.15.86:0/116890432 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8040cd4b0 con 0x7fe8040d08c0 2023-06-23T00:39:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe8037fe700 1 --2- 172.21.15.86:0/116890432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8040d08c0 0x7fe80415a410 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc009e60 tx=0x7fe7fc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c5b1b6b315f02d9 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe801ffb700 1 -- 172.21.15.86:0/116890432 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc00e070 con 0x7fe8040d08c0 2023-06-23T00:39:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe801ffb700 1 -- 172.21.15.86:0/116890432 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7fc00bab0 con 0x7fe8040d08c0 2023-06-23T00:39:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe801ffb700 1 -- 172.21.15.86:0/116890432 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc00be70 con 0x7fe8040d08c0 2023-06-23T00:39:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe809312700 1 -- 172.21.15.86:0/116890432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8040d08c0 msgr2=0x7fe80415a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe809312700 1 --2- 172.21.15.86:0/116890432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8040d08c0 0x7fe80415a410 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc009e60 tx=0x7fe7fc009780 comp rx=0 tx=0).stop 2023-06-23T00:39:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe809312700 1 -- 172.21.15.86:0/116890432 shutdown_connections 2023-06-23T00:39:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe809312700 1 --2- 172.21.15.86:0/116890432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8040d08c0 0x7fe80415a410 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe809312700 1 -- 172.21.15.86:0/116890432 >> 172.21.15.86:0/116890432 conn(0x7fe80406a060 msgr2=0x7fe80406a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe809312700 1 -- 172.21.15.86:0/116890432 shutdown_connections 2023-06-23T00:39:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.272+0000 7fe809312700 1 -- 172.21.15.86:0/116890432 wait complete. 2023-06-23T00:39:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.274+0000 7fe809312700 1 Processor -- start 2023-06-23T00:39:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.274+0000 7fe809312700 1 -- start start 2023-06-23T00:39:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.274+0000 7fe809312700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8040d04f0 0x7fe8040ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.274+0000 7fe809312700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8040d08c0 0x7fe80415e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.274+0000 7fe809312700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8041570c0 0x7fe8041528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.274+0000 7fe809312700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8040b4890 con 0x7fe8040d04f0 2023-06-23T00:39:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.274+0000 7fe809312700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8040b4b30 con 0x7fe8040d08c0 2023-06-23T00:39:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.274+0000 7fe809312700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80415ec90 con 0x7fe8041570c0 2023-06-23T00:39:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe8027fc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8040d08c0 0x7fe80415e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe8037fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8041570c0 0x7fe8041528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe8027fc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8040d08c0 0x7fe80415e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38544/0 (socket says 172.21.15.86:38544) 2023-06-23T00:39:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe8027fc700 1 -- 172.21.15.86:0/1621028062 learned_addr learned my addr 172.21.15.86:0/1621028062 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-06-23T00:39:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe802ffd700 1 --2- 172.21.15.86:0/1621028062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8040d04f0 0x7fe8040ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe8037fe700 1 -- 172.21.15.86:0/1621028062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8040d08c0 msgr2=0x7fe80415e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe8037fe700 1 --2- 172.21.15.86:0/1621028062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8040d08c0 0x7fe80415e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe8037fe700 1 -- 172.21.15.86:0/1621028062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8040d04f0 msgr2=0x7fe8040ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe8037fe700 1 --2- 172.21.15.86:0/1621028062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8040d04f0 0x7fe8040ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe8037fe700 1 -- 172.21.15.86:0/1621028062 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc009a20 con 0x7fe8041570c0 2023-06-23T00:39:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe8027fc700 1 --2- 172.21.15.86:0/1621028062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8040d08c0 0x7fe80415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:39:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe8037fe700 1 --2- 172.21.15.86:0/1621028062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8041570c0 0x7fe8041528a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc016000 tx=0x7fe7fc019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe7eaffd700 1 -- 172.21.15.86:0/1621028062 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc012dd0 con 0x7fe8041570c0 2023-06-23T00:39:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe7eaffd700 1 -- 172.21.15.86:0/1621028062 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7fc025430 con 0x7fe8041570c0 2023-06-23T00:39:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe809312700 1 -- 172.21.15.86:0/1621028062 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80415e070 con 0x7fe8041570c0 2023-06-23T00:39:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe7eaffd700 1 -- 172.21.15.86:0/1621028062 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc0257f0 con 0x7fe8041570c0 2023-06-23T00:39:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe809312700 1 -- 172.21.15.86:0/1621028062 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80415dd90 con 0x7fe8041570c0 2023-06-23T00:39:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe7eaffd700 1 -- 172.21.15.86:0/1621028062 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe7fc025ba0 con 0x7fe8041570c0 2023-06-23T00:39:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.275+0000 7fe7eaffd700 1 -- 172.21.15.86:0/1621028062 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7fe7fc01ca10 con 0x7fe8041570c0 2023-06-23T00:39:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.276+0000 7fe809312700 1 -- 172.21.15.86:0/1621028062 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80404c1b0 con 0x7fe8041570c0 2023-06-23T00:39:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.279+0000 7fe7eaffd700 1 -- 172.21.15.86:0/1621028062 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fe7fc02a020 con 0x7fe8041570c0 2023-06-23T00:39:59.317 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:39:59.314+0000 7f84c915b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T00:39:59.336 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:39:59.332+0000 7fbcd7ef5180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T00:39:59.376 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.372+0000 7fe809312700 1 -- 172.21.15.86:0/1621028062 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a5ece869-9da7-40e8-8f1d-bbaf67bb2187", "id": 2} v 0) v1 -- 0x7fe8040cd790 con 0x7fe8041570c0 2023-06-23T00:39:59.379 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.376+0000 7fe7eaffd700 1 -- 172.21.15.86:0/1621028062 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a5ece869-9da7-40e8-8f1d-bbaf67bb2187", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe7fc01cc70 con 0x7fe8041570c0 2023-06-23T00:39:59.380 INFO:teuthology.orchestra.run.smithi086.stdout:2 2023-06-23T00:39:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.377+0000 7fe809312700 1 -- 172.21.15.86:0/1621028062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8041570c0 msgr2=0x7fe8041528a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.377+0000 7fe809312700 1 --2- 172.21.15.86:0/1621028062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8041570c0 0x7fe8041528a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc016000 tx=0x7fe7fc019000 comp rx=0 tx=0).stop 2023-06-23T00:39:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.377+0000 7fe809312700 1 -- 172.21.15.86:0/1621028062 shutdown_connections 2023-06-23T00:39:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.377+0000 7fe809312700 1 --2- 172.21.15.86:0/1621028062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8041570c0 0x7fe8041528a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.377+0000 7fe809312700 1 --2- 172.21.15.86:0/1621028062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe8040d04f0 0x7fe8040ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.377+0000 7fe809312700 1 --2- 172.21.15.86:0/1621028062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8040d08c0 0x7fe80415e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.377+0000 7fe809312700 1 -- 172.21.15.86:0/1621028062 >> 172.21.15.86:0/1621028062 conn(0x7fe80406a060 msgr2=0x7fe804154700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.377+0000 7fe809312700 1 -- 172.21.15.86:0/1621028062 shutdown_connections 2023-06-23T00:39:59.382 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.377+0000 7fe809312700 1 -- 172.21.15.86:0/1621028062 wait complete. 2023-06-23T00:39:59.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 21067fac-663b-4ecf-ae67-bf235161638e 3 2023-06-23T00:39:59.457 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:39:59.454+0000 7f84c915b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T00:39:59.474 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:39:59.471+0000 7fbcd7ef5180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T00:39:59.509 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.506+0000 7f3ca6710700 1 Processor -- start 2023-06-23T00:39:59.510 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.506+0000 7f3ca6710700 1 -- start start 2023-06-23T00:39:59.511 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca6710700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ca0140a50 0x7f3ca0068f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca6710700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3ca00aea50 0x7f3ca00cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca6710700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 0x7f3ca00b2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca6710700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca00b2a50 con 0x7f3ca00aea50 2023-06-23T00:39:59.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca6710700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca0157ad0 con 0x7f3ca0140a50 2023-06-23T00:39:59.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca6710700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca0157db0 con 0x7f3ca00b2070 2023-06-23T00:39:59.512 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3c9ffff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ca0140a50 0x7f3ca0068f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca4cad700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 0x7f3ca00b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3c9ffff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ca0140a50 0x7f3ca0068f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38554/0 (socket says 172.21.15.86:38554) 2023-06-23T00:39:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3c9ffff700 1 -- 172.21.15.86:0/2304191001 learned_addr learned my addr 172.21.15.86:0/2304191001 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-06-23T00:39:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3c9f7fe700 1 --2- 172.21.15.86:0/2304191001 >> v2:172.21.15.64:3300/0 conn(0x7f3ca00aea50 0x7f3ca00cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca4cad700 1 -- 172.21.15.86:0/2304191001 >> v2:172.21.15.86:3300/0 conn(0x7f3ca0140a50 msgr2=0x7f3ca0068f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:39:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca4cad700 1 --2- 172.21.15.86:0/2304191001 >> v2:172.21.15.86:3300/0 conn(0x7f3ca0140a50 0x7f3ca0068f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca4cad700 1 -- 172.21.15.86:0/2304191001 >> v2:172.21.15.64:3300/0 conn(0x7f3ca00aea50 msgr2=0x7f3ca00cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:39:59.513 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca4cad700 1 --2- 172.21.15.86:0/2304191001 >> v2:172.21.15.64:3300/0 conn(0x7f3ca00aea50 0x7f3ca00cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca4cad700 1 -- 172.21.15.86:0/2304191001 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca0158090 con 0x7f3ca00b2070 2023-06-23T00:39:59.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3c9ffff700 1 --2- 172.21.15.86:0/2304191001 >> v2:172.21.15.86:3300/0 conn(0x7f3ca0140a50 0x7f3ca0068f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:39:59.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.507+0000 7f3ca4cad700 1 --2- 172.21.15.86:0/2304191001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 0x7f3ca00b2510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3ca006dc10 tx=0x7f3c98009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3eb38135fb3fc79 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:59.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3c9effd700 1 -- 172.21.15.86:0/2304191001 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9800e070 con 0x7f3ca00b2070 2023-06-23T00:39:59.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3c9effd700 1 -- 172.21.15.86:0/2304191001 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9800bab0 con 0x7f3ca00b2070 2023-06-23T00:39:59.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3c9effd700 1 -- 172.21.15.86:0/2304191001 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9800be50 con 0x7f3ca00b2070 2023-06-23T00:39:59.514 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3ca6710700 1 -- 172.21.15.86:0/2304191001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 msgr2=0x7f3ca00b2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3ca6710700 1 --2- 172.21.15.86:0/2304191001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 0x7f3ca00b2510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3ca006dc10 tx=0x7f3c98009780 comp rx=0 tx=0).stop 2023-06-23T00:39:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3ca6710700 1 -- 172.21.15.86:0/2304191001 shutdown_connections 2023-06-23T00:39:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3ca6710700 1 --2- 172.21.15.86:0/2304191001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 0x7f3ca00b2510 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3ca6710700 1 --2- 172.21.15.86:0/2304191001 >> v2:172.21.15.86:3300/0 conn(0x7f3ca0140a50 0x7f3ca0068f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3ca6710700 1 --2- 172.21.15.86:0/2304191001 >> v2:172.21.15.64:3300/0 conn(0x7f3ca00aea50 0x7f3ca00cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3ca6710700 1 -- 172.21.15.86:0/2304191001 >> 172.21.15.86:0/2304191001 conn(0x7f3ca006a0c0 msgr2=0x7f3ca006b100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3ca6710700 1 -- 172.21.15.86:0/2304191001 shutdown_connections 2023-06-23T00:39:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.508+0000 7f3ca6710700 1 -- 172.21.15.86:0/2304191001 wait complete. 2023-06-23T00:39:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.509+0000 7f3ca6710700 1 Processor -- start 2023-06-23T00:39:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.509+0000 7f3ca6710700 1 -- start start 2023-06-23T00:39:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.510+0000 7f3ca6710700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ca00aea50 0x7f3ca016a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.510+0000 7f3ca6710700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 0x7f3ca01676c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.510+0000 7f3ca6710700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca0140a50 0x7f3ca015d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.510+0000 7f3ca6710700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca0167230 con 0x7f3ca00aea50 2023-06-23T00:39:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.510+0000 7f3ca6710700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca016ac90 con 0x7f3ca0140a50 2023-06-23T00:39:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.510+0000 7f3ca6710700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca016af70 con 0x7f3ca00b2070 2023-06-23T00:39:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.510+0000 7f3c9f7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 0x7f3ca01676c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.510+0000 7f3ca4cad700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca0140a50 0x7f3ca015d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.510+0000 7f3c9f7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 0x7f3ca01676c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48354/0 (socket says 172.21.15.86:48354) 2023-06-23T00:39:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.510+0000 7f3c9f7fe700 1 -- 172.21.15.86:0/3466059070 learned_addr learned my addr 172.21.15.86:0/3466059070 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-06-23T00:39:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.510+0000 7f3c9ffff700 1 --2- 172.21.15.86:0/3466059070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ca00aea50 0x7f3ca016a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3c9f7fe700 1 -- 172.21.15.86:0/3466059070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca0140a50 msgr2=0x7f3ca015d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3c9f7fe700 1 --2- 172.21.15.86:0/3466059070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca0140a50 0x7f3ca015d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3c9f7fe700 1 -- 172.21.15.86:0/3466059070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ca00aea50 msgr2=0x7f3ca016a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3c9f7fe700 1 --2- 172.21.15.86:0/3466059070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ca00aea50 0x7f3ca016a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3c9f7fe700 1 -- 172.21.15.86:0/3466059070 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c900094f0 con 0x7f3ca00b2070 2023-06-23T00:39:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3ca4cad700 1 --2- 172.21.15.86:0/3466059070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca0140a50 0x7f3ca015d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:39:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3c9f7fe700 1 --2- 172.21.15.86:0/3466059070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 0x7f3ca01676c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400ea50 tx=0x7f3c940091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3c87fff700 1 -- 172.21.15.86:0/3466059070 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400ee90 con 0x7f3ca00b2070 2023-06-23T00:39:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3c87fff700 1 -- 172.21.15.86:0/3466059070 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9400c850 con 0x7f3ca00b2070 2023-06-23T00:39:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3ca6710700 1 -- 172.21.15.86:0/3466059070 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c98009a20 con 0x7f3ca00b2070 2023-06-23T00:39:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3c87fff700 1 -- 172.21.15.86:0/3466059070 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400cbf0 con 0x7f3ca00b2070 2023-06-23T00:39:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3ca6710700 1 -- 172.21.15.86:0/3466059070 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca0166ec0 con 0x7f3ca00b2070 2023-06-23T00:39:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3c87fff700 1 -- 172.21.15.86:0/3466059070 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3c94005c80 con 0x7f3ca00b2070 2023-06-23T00:39:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.511+0000 7f3c87fff700 1 -- 172.21.15.86:0/3466059070 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f3c94013470 con 0x7f3ca00b2070 2023-06-23T00:39:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.512+0000 7f3ca6710700 1 -- 172.21.15.86:0/3466059070 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca004c1b0 con 0x7f3ca00b2070 2023-06-23T00:39:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.514+0000 7f3c87fff700 1 -- 172.21.15.86:0/3466059070 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f3c94021060 con 0x7f3ca00b2070 2023-06-23T00:39:59.535 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:39:59.532+0000 7f84c915b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T00:39:59.554 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:39:59.551+0000 7fbcd7ef5180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T00:39:59.601 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.597+0000 7f3ca6710700 1 -- 172.21.15.86:0/3466059070 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "21067fac-663b-4ecf-ae67-bf235161638e", "id": 3} v 0) v1 -- 0x7f3ca0158370 con 0x7f3ca00b2070 2023-06-23T00:39:59.604 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.601+0000 7f3c87fff700 1 -- 172.21.15.86:0/3466059070 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "21067fac-663b-4ecf-ae67-bf235161638e", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3c94017070 con 0x7f3ca00b2070 2023-06-23T00:39:59.605 INFO:teuthology.orchestra.run.smithi086.stdout:3 2023-06-23T00:39:59.605 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.602+0000 7f3ca6710700 1 -- 172.21.15.86:0/3466059070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 msgr2=0x7f3ca01676c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.602+0000 7f3ca6710700 1 --2- 172.21.15.86:0/3466059070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 0x7f3ca01676c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3c9400ea50 tx=0x7f3c940091a0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.602+0000 7f3ca6710700 1 -- 172.21.15.86:0/3466059070 shutdown_connections 2023-06-23T00:39:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.602+0000 7f3ca6710700 1 --2- 172.21.15.86:0/3466059070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ca0140a50 0x7f3ca015d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.602+0000 7f3ca6710700 1 --2- 172.21.15.86:0/3466059070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ca00aea50 0x7f3ca016a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.602+0000 7f3ca6710700 1 --2- 172.21.15.86:0/3466059070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ca00b2070 0x7f3ca01676c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.602+0000 7f3ca6710700 1 -- 172.21.15.86:0/3466059070 >> 172.21.15.86:0/3466059070 conn(0x7f3ca006a0c0 msgr2=0x7f3ca00adae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:59.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.602+0000 7f3ca6710700 1 -- 172.21.15.86:0/3466059070 shutdown_connections 2023-06-23T00:39:59.607 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.602+0000 7f3ca6710700 1 -- 172.21.15.86:0/3466059070 wait complete. 2023-06-23T00:39:59.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd 4 2023-06-23T00:39:59.726 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:39:59.722+0000 7f84c915b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T00:39:59.732 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.729+0000 7fd2b6bc6700 1 Processor -- start 2023-06-23T00:39:59.733 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.729+0000 7fd2b6bc6700 1 -- start start 2023-06-23T00:39:59.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.730+0000 7fd2b6bc6700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fd2b00b2160 0x7fd2b006a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.734 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.730+0000 7fd2b6bc6700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b006a1b0 con 0x7fd2b00aefa0 2023-06-23T00:39:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.730+0000 7fd2b6bc6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b00b2740 con 0x7fd2b006ab80 2023-06-23T00:39:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.730+0000 7fd2b6bc6700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b0069b60 con 0x7fd2b00b2160 2023-06-23T00:39:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.730+0000 7fd2b63c5700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd2b006ab80 0x7fd2b00ca640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40172/0 (socket says 172.21.15.86:40172) 2023-06-23T00:39:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.730+0000 7fd2b63c5700 1 -- 172.21.15.86:0/296171459 learned_addr learned my addr 172.21.15.86:0/296171459 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-06-23T00:39:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.730+0000 7fd2b53c3700 1 --2- 172.21.15.86:0/296171459 >> v2:172.21.15.64:3300/0 conn(0x7fd2b00b2160 0x7fd2b006a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.730+0000 7fd2b4bc2700 1 -- 172.21.15.86:0/296171459 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655327763 0 0) 0x7fd2b00b2740 con 0x7fd2b006ab80 2023-06-23T00:39:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.730+0000 7fd2b4bc2700 1 -- 172.21.15.86:0/296171459 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd29c003570 con 0x7fd2b006ab80 2023-06-23T00:39:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.730+0000 7fd2b53c3700 1 -- 172.21.15.86:0/296171459 >> v1:172.21.15.86:6789/0 conn(0x7fd2b006ab80 legacy=0x7fd2b00ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.730+0000 7fd2b53c3700 1 -- 172.21.15.86:0/296171459 >> v1:172.21.15.64:6789/0 conn(0x7fd2b00aefa0 legacy=0x7fd2b00b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.731+0000 7fd2b53c3700 1 -- 172.21.15.86:0/296171459 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b0069e40 con 0x7fd2b00b2160 2023-06-23T00:39:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.731+0000 7fd2b53c3700 1 --2- 172.21.15.86:0/296171459 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2b00b2160 0x7fd2b006a640 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0098e0 tx=0x7fd2ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d70dd3663d3872a server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.731+0000 7fd2b4bc2700 1 -- 172.21.15.86:0/296171459 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac011630 con 0x7fd2b00b2160 2023-06-23T00:39:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.731+0000 7fd2b4bc2700 1 -- 172.21.15.86:0/296171459 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac011790 con 0x7fd2b00b2160 2023-06-23T00:39:59.736 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.731+0000 7fd2b4bc2700 1 -- 172.21.15.86:0/296171459 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac011b30 con 0x7fd2b00b2160 2023-06-23T00:39:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.731+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/296171459 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2b00b2160 msgr2=0x7fd2b006a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.731+0000 7fd2b6bc6700 1 --2- 172.21.15.86:0/296171459 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2b00b2160 0x7fd2b006a640 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0098e0 tx=0x7fd2ac0091a0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.731+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/296171459 shutdown_connections 2023-06-23T00:39:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.731+0000 7fd2b6bc6700 1 --2- 172.21.15.86:0/296171459 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2b00b2160 0x7fd2b006a640 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.731+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/296171459 >> 172.21.15.86:0/296171459 conn(0x7fd2b00d0b70 msgr2=0x7fd2b00ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.732+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/296171459 shutdown_connections 2023-06-23T00:39:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.732+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/296171459 wait complete. 2023-06-23T00:39:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.733+0000 7fd2b6bc6700 1 Processor -- start 2023-06-23T00:39:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.733+0000 7fd2b6bc6700 1 -- start start 2023-06-23T00:39:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.733+0000 7fd2b6bc6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2b006ab80 0x7fd2b0159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.733+0000 7fd2b6bc6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2b00aefa0 0x7fd2b016b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.733+0000 7fd2b6bc6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2b00b2160 0x7fd2b0169840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.740 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.733+0000 7fd2b6bc6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b0158870 con 0x7fd2b006ab80 2023-06-23T00:39:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.733+0000 7fd2b6bc6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b0158b50 con 0x7fd2b00aefa0 2023-06-23T00:39:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.733+0000 7fd2b6bc6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b0158e30 con 0x7fd2b00b2160 2023-06-23T00:39:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.733+0000 7fd2b53c3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2b00aefa0 0x7fd2b016b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.733+0000 7fd2b63c5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2b00b2160 0x7fd2b0169840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2b53c3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2b00aefa0 0x7fd2b016b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38574/0 (socket says 172.21.15.86:38574) 2023-06-23T00:39:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2b53c3700 1 -- 172.21.15.86:0/1028632446 learned_addr learned my addr 172.21.15.86:0/1028632446 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-06-23T00:39:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2b5bc4700 1 --2- 172.21.15.86:0/1028632446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2b006ab80 0x7fd2b0159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.741 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2b63c5700 1 -- 172.21.15.86:0/1028632446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2b00aefa0 msgr2=0x7fd2b016b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2b63c5700 1 --2- 172.21.15.86:0/1028632446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2b00aefa0 0x7fd2b016b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2b63c5700 1 -- 172.21.15.86:0/1028632446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2b006ab80 msgr2=0x7fd2b0159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.742 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2b63c5700 1 --2- 172.21.15.86:0/1028632446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2b006ab80 0x7fd2b0159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2b63c5700 1 -- 172.21.15.86:0/1028632446 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac009420 con 0x7fd2b00b2160 2023-06-23T00:39:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2b63c5700 1 --2- 172.21.15.86:0/1028632446 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2b00b2160 0x7fd2b0169840 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800cbc0 tx=0x7fd2a8001a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:59.743 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2a5ffb700 1 -- 172.21.15.86:0/1028632446 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a800a7e0 con 0x7fd2b00b2160 2023-06-23T00:39:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2a5ffb700 1 -- 172.21.15.86:0/1028632446 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a800a940 con 0x7fd2b00b2160 2023-06-23T00:39:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/1028632446 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b0159110 con 0x7fd2b00b2160 2023-06-23T00:39:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2a5ffb700 1 -- 172.21.15.86:0/1028632446 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a800acc0 con 0x7fd2b00b2160 2023-06-23T00:39:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/1028632446 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b016b670 con 0x7fd2b00b2160 2023-06-23T00:39:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.734+0000 7fd2a5ffb700 1 -- 172.21.15.86:0/1028632446 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd2a8007300 con 0x7fd2b00b2160 2023-06-23T00:39:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.735+0000 7fd2a5ffb700 1 -- 172.21.15.86:0/1028632446 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fd2a801b910 con 0x7fd2b00b2160 2023-06-23T00:39:59.744 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.736+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/1028632446 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b004c1b0 con 0x7fd2b00b2160 2023-06-23T00:39:59.745 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.738+0000 7fd2a5ffb700 1 -- 172.21.15.86:0/1028632446 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fd2a8007790 con 0x7fd2b00b2160 2023-06-23T00:39:59.749 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:39:59.746+0000 7fbcd7ef5180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T00:39:59.823 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.819+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/1028632446 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd", "id": 4} v 0) v1 -- 0x7fd2b015b0c0 con 0x7fd2b00b2160 2023-06-23T00:39:59.827 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.823+0000 7fd2a5ffb700 1 -- 172.21.15.86:0/1028632446 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd2a8007930 con 0x7fd2b00b2160 2023-06-23T00:39:59.827 INFO:teuthology.orchestra.run.smithi086.stdout:4 2023-06-23T00:39:59.828 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.824+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/1028632446 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2b00b2160 msgr2=0x7fd2b0169840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.829 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.824+0000 7fd2b6bc6700 1 --2- 172.21.15.86:0/1028632446 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2b00b2160 0x7fd2b0169840 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800cbc0 tx=0x7fd2a8001a20 comp rx=0 tx=0).stop 2023-06-23T00:39:59.829 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.825+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/1028632446 shutdown_connections 2023-06-23T00:39:59.829 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.825+0000 7fd2b6bc6700 1 --2- 172.21.15.86:0/1028632446 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2b00b2160 0x7fd2b0169840 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.830 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.825+0000 7fd2b6bc6700 1 --2- 172.21.15.86:0/1028632446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2b006ab80 0x7fd2b0159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.830 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.825+0000 7fd2b6bc6700 1 --2- 172.21.15.86:0/1028632446 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2b00aefa0 0x7fd2b016b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.830 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.825+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/1028632446 >> 172.21.15.86:0/1028632446 conn(0x7fd2b00d0b70 msgr2=0x7fd2b0154970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:59.830 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.825+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/1028632446 shutdown_connections 2023-06-23T00:39:59.830 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.825+0000 7fd2b6bc6700 1 -- 172.21.15.86:0/1028632446 wait complete. 2023-06-23T00:39:59.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 0e987a73-8b36-4f26-94fd-f0135d6062c8 5 2023-06-23T00:39:59.906 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:39:59.902+0000 7f84c915b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T00:39:59.929 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:39:59.926+0000 7fbcd7ef5180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T00:39:59.954 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.951+0000 7f8adb220700 1 Processor -- start 2023-06-23T00:39:59.955 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.951+0000 7f8adb220700 1 -- start start 2023-06-23T00:39:59.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8adb220700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40ccca0 0x7f8ad4164360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.956 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8adb220700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad406e360 con 0x7f8ad40b0450 2023-06-23T00:39:59.957 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8adb220700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad40c9b60 con 0x7f8ad40c9680 2023-06-23T00:39:59.957 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8adb220700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad4159970 con 0x7f8ad40ccca0 2023-06-23T00:39:59.957 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8ad97bd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40ccca0 0x7f8ad4164360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.957 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8ad97bd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40ccca0 0x7f8ad4164360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48380/0 (socket says 172.21.15.86:48380) 2023-06-23T00:39:59.957 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8ad97bd700 1 -- 172.21.15.86:0/766317769 learned_addr learned my addr 172.21.15.86:0/766317769 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-06-23T00:39:59.957 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8ad3fff700 1 --1- 172.21.15.86:0/766317769 >> v1:172.21.15.86:6789/0 conn(0x7f8ad40c9680 0x7f8ad40cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40182/0 (socket says 172.21.15.86:40182) 2023-06-23T00:39:59.958 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8ad97bd700 1 -- 172.21.15.86:0/766317769 >> v1:172.21.15.86:6789/0 conn(0x7f8ad40c9680 legacy=0x7f8ad40cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.958 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8ad97bd700 1 -- 172.21.15.86:0/766317769 >> v1:172.21.15.64:6789/0 conn(0x7f8ad40b0450 legacy=0x7f8ad406dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.958 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8ad37fe700 1 -- 172.21.15.86:0/766317769 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839316182 0 0) 0x7f8ad40c9b60 con 0x7f8ad40c9680 2023-06-23T00:39:59.958 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8ad97bd700 1 -- 172.21.15.86:0/766317769 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad40c9e40 con 0x7f8ad40ccca0 2023-06-23T00:39:59.958 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8ad97bd700 1 --2- 172.21.15.86:0/766317769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40ccca0 0x7f8ad4164360 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009f50 tx=0x7f8acc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=515c1ee1286f6a17 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:59.958 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.952+0000 7f8ad37fe700 1 -- 172.21.15.86:0/766317769 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc00e070 con 0x7f8ad40ccca0 2023-06-23T00:39:59.958 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.953+0000 7f8ad37fe700 1 -- 172.21.15.86:0/766317769 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc00b9b0 con 0x7f8ad40ccca0 2023-06-23T00:39:59.959 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.953+0000 7f8ad37fe700 1 -- 172.21.15.86:0/766317769 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc0092e0 con 0x7f8ad40ccca0 2023-06-23T00:39:59.959 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.953+0000 7f8adb220700 1 -- 172.21.15.86:0/766317769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40ccca0 msgr2=0x7f8ad4164360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.959 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.953+0000 7f8adb220700 1 --2- 172.21.15.86:0/766317769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40ccca0 0x7f8ad4164360 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009f50 tx=0x7f8acc009840 comp rx=0 tx=0).stop 2023-06-23T00:39:59.964 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.953+0000 7f8adb220700 1 -- 172.21.15.86:0/766317769 shutdown_connections 2023-06-23T00:39:59.965 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.953+0000 7f8adb220700 1 --2- 172.21.15.86:0/766317769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40ccca0 0x7f8ad4164360 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.965 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.953+0000 7f8adb220700 1 -- 172.21.15.86:0/766317769 >> 172.21.15.86:0/766317769 conn(0x7f8ad406cb90 msgr2=0x7f8ad40ce050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:39:59.965 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.953+0000 7f8adb220700 1 -- 172.21.15.86:0/766317769 shutdown_connections 2023-06-23T00:39:59.965 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.953+0000 7f8adb220700 1 -- 172.21.15.86:0/766317769 wait complete. 2023-06-23T00:39:59.965 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.954+0000 7f8adb220700 1 Processor -- start 2023-06-23T00:39:59.965 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8adb220700 1 -- start start 2023-06-23T00:39:59.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8adb220700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ad40b0450 0x7f8ad415b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8adb220700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40c9680 0x7f8ad4151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8adb220700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad40ccca0 0x7f8ad4157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:39:59.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8adb220700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad415e5d0 con 0x7f8ad40b0450 2023-06-23T00:39:59.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8adb220700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad415b6e0 con 0x7f8ad40ccca0 2023-06-23T00:39:59.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8adb220700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad415b910 con 0x7f8ad40c9680 2023-06-23T00:39:59.966 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8ad3fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40c9680 0x7f8ad4151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8ad97bd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad40ccca0 0x7f8ad4157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8ad3fff700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40c9680 0x7f8ad4151b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48388/0 (socket says 172.21.15.86:48388) 2023-06-23T00:39:59.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8ad97bd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad40ccca0 0x7f8ad4157cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38586/0 (socket says 172.21.15.86:38586) 2023-06-23T00:39:59.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8ad3fff700 1 -- 172.21.15.86:0/939235276 learned_addr learned my addr 172.21.15.86:0/939235276 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-06-23T00:39:59.967 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.955+0000 7f8ad8fbc700 1 --2- 172.21.15.86:0/939235276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ad40b0450 0x7f8ad415b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:39:59.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad97bd700 1 -- 172.21.15.86:0/939235276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40c9680 msgr2=0x7f8ad4151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad97bd700 1 --2- 172.21.15.86:0/939235276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40c9680 0x7f8ad4151b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad97bd700 1 -- 172.21.15.86:0/939235276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ad40b0450 msgr2=0x7f8ad415b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:39:59.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad97bd700 1 --2- 172.21.15.86:0/939235276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ad40b0450 0x7f8ad415b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:39:59.968 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad97bd700 1 -- 172.21.15.86:0/939235276 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8acc009ae0 con 0x7f8ad40ccca0 2023-06-23T00:39:59.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad3fff700 1 --2- 172.21.15.86:0/939235276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40c9680 0x7f8ad4151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:39:59.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad97bd700 1 --2- 172.21.15.86:0/939235276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad40ccca0 0x7f8ad4157cf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8acc014000 tx=0x7f8acc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:39:59.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad0ff9700 1 -- 172.21.15.86:0/939235276 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc012bc0 con 0x7f8ad40ccca0 2023-06-23T00:39:59.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad0ff9700 1 -- 172.21.15.86:0/939235276 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc012d60 con 0x7f8ad40ccca0 2023-06-23T00:39:59.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad0ff9700 1 -- 172.21.15.86:0/939235276 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc01d670 con 0x7f8ad40ccca0 2023-06-23T00:39:59.969 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8adb220700 1 -- 172.21.15.86:0/939235276 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad415bbf0 con 0x7f8ad40ccca0 2023-06-23T00:39:59.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8adb220700 1 -- 172.21.15.86:0/939235276 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad415c1b0 con 0x7f8ad40ccca0 2023-06-23T00:39:59.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad0ff9700 1 -- 172.21.15.86:0/939235276 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8acc01daa0 con 0x7f8ad40ccca0 2023-06-23T00:39:59.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.956+0000 7f8ad0ff9700 1 -- 172.21.15.86:0/939235276 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f8acc025a10 con 0x7f8ad40ccca0 2023-06-23T00:39:59.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.957+0000 7f8adb220700 1 -- 172.21.15.86:0/939235276 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad40c5100 con 0x7f8ad40ccca0 2023-06-23T00:39:59.970 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:39:59.959+0000 7f8ad0ff9700 1 -- 172.21.15.86:0/939235276 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f8ad40c5100 con 0x7f8ad40ccca0 2023-06-23T00:39:59.987 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:39:59.984+0000 7f84c915b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T00:40:00.009 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:00.006+0000 7fbcd7ef5180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T00:40:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.042+0000 7f8adb220700 1 -- 172.21.15.86:0/939235276 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0e987a73-8b36-4f26-94fd-f0135d6062c8", "id": 5} v 0) v1 -- 0x7f8ad404c1b0 con 0x7f8ad40ccca0 2023-06-23T00:40:00.049 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.046+0000 7f8ad0ff9700 1 -- 172.21.15.86:0/939235276 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0e987a73-8b36-4f26-94fd-f0135d6062c8", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8acc02a060 con 0x7f8ad40ccca0 2023-06-23T00:40:00.049 INFO:teuthology.orchestra.run.smithi086.stdout:5 2023-06-23T00:40:00.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.047+0000 7f8adb220700 1 -- 172.21.15.86:0/939235276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad40ccca0 msgr2=0x7f8ad4157cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.050 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.047+0000 7f8adb220700 1 --2- 172.21.15.86:0/939235276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad40ccca0 0x7f8ad4157cf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8acc014000 tx=0x7f8acc017000 comp rx=0 tx=0).stop 2023-06-23T00:40:00.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.047+0000 7f8adb220700 1 -- 172.21.15.86:0/939235276 shutdown_connections 2023-06-23T00:40:00.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.047+0000 7f8adb220700 1 --2- 172.21.15.86:0/939235276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ad40ccca0 0x7f8ad4157cf0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.047+0000 7f8adb220700 1 --2- 172.21.15.86:0/939235276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ad40b0450 0x7f8ad415b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.047+0000 7f8adb220700 1 --2- 172.21.15.86:0/939235276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ad40c9680 0x7f8ad4151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.047+0000 7f8adb220700 1 -- 172.21.15.86:0/939235276 >> 172.21.15.86:0/939235276 conn(0x7f8ad406cb90 msgr2=0x7f8ad40c2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:00.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.047+0000 7f8adb220700 1 -- 172.21.15.86:0/939235276 shutdown_connections 2023-06-23T00:40:00.051 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.047+0000 7f8adb220700 1 -- 172.21.15.86:0/939235276 wait complete. 2023-06-23T00:40:00.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 7e40bfd5-0f9f-448c-8ccf-2f44bf82a622 6 2023-06-23T00:40:00.141 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:00.138+0000 7f84c915b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T00:40:00.149 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:00.146+0000 7fbcd7ef5180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T00:40:00.180 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.177+0000 7fc7c189e700 1 Processor -- start 2023-06-23T00:40:00.181 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.177+0000 7fc7c189e700 1 -- start start 2023-06-23T00:40:00.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7c189e700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fc7bc1555b0 0x7fc7bc06e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7c189e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc155980 0x7fc7bc0b0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7c189e700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7bc0b0fb0 con 0x7fc7bc0b0b50 2023-06-23T00:40:00.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7c189e700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7bc069f80 con 0x7fc7bc1555b0 2023-06-23T00:40:00.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7c189e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc06ada0 con 0x7fc7bc155980 2023-06-23T00:40:00.183 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7bb7fe700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc7bc0b0b50 0x7fc7bc1593e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40186/0 (socket says 172.21.15.86:40186) 2023-06-23T00:40:00.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7bb7fe700 1 -- 172.21.15.86:0/2028899926 learned_addr learned my addr 172.21.15.86:0/2028899926 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-06-23T00:40:00.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7ba7fc700 1 --2- 172.21.15.86:0/2028899926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc155980 0x7fc7bc0b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7baffd700 1 --2- 172.21.15.86:0/2028899926 >> v2:172.21.15.64:3300/0 conn(0x7fc7bc1555b0 0x7fc7bc06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7b9ffb700 1 -- 172.21.15.86:0/2028899926 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281829094 0 0) 0x7fc7ac001110 con 0x7fc7bc0b0b50 2023-06-23T00:40:00.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7b9ffb700 1 -- 172.21.15.86:0/2028899926 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a4003570 con 0x7fc7bc0b0b50 2023-06-23T00:40:00.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7ba7fc700 1 -- 172.21.15.86:0/2028899926 >> v2:172.21.15.64:3300/0 conn(0x7fc7bc1555b0 msgr2=0x7fc7bc06e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:00.184 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7ba7fc700 1 --2- 172.21.15.86:0/2028899926 >> v2:172.21.15.64:3300/0 conn(0x7fc7bc1555b0 0x7fc7bc06e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7ba7fc700 1 -- 172.21.15.86:0/2028899926 >> v1:172.21.15.86:6789/0 conn(0x7fc7bc0b0b50 legacy=0x7fc7bc1593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7ba7fc700 1 -- 172.21.15.86:0/2028899926 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc06eec0 con 0x7fc7bc155980 2023-06-23T00:40:00.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.178+0000 7fc7ba7fc700 1 --2- 172.21.15.86:0/2028899926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc155980 0x7fc7bc0b0610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc7b000aa90 tx=0x7fc7b000a330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68b12f32ee607160 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:00.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.179+0000 7fc7b9ffb700 1 -- 172.21.15.86:0/2028899926 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b000c970 con 0x7fc7bc155980 2023-06-23T00:40:00.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.179+0000 7fc7b9ffb700 1 -- 172.21.15.86:0/2028899926 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b000cad0 con 0x7fc7bc155980 2023-06-23T00:40:00.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.179+0000 7fc7b9ffb700 1 -- 172.21.15.86:0/2028899926 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b000ce70 con 0x7fc7bc155980 2023-06-23T00:40:00.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.179+0000 7fc7c189e700 1 -- 172.21.15.86:0/2028899926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc155980 msgr2=0x7fc7bc0b0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.185 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.179+0000 7fc7c189e700 1 --2- 172.21.15.86:0/2028899926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc155980 0x7fc7bc0b0610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc7b000aa90 tx=0x7fc7b000a330 comp rx=0 tx=0).stop 2023-06-23T00:40:00.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.179+0000 7fc7c189e700 1 -- 172.21.15.86:0/2028899926 shutdown_connections 2023-06-23T00:40:00.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.179+0000 7fc7c189e700 1 --2- 172.21.15.86:0/2028899926 >> v2:172.21.15.64:3300/0 conn(0x7fc7bc1555b0 0x7fc7bc06e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.179+0000 7fc7c189e700 1 --2- 172.21.15.86:0/2028899926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc155980 0x7fc7bc0b0610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.179+0000 7fc7c189e700 1 -- 172.21.15.86:0/2028899926 >> 172.21.15.86:0/2028899926 conn(0x7fc7bc068fb0 msgr2=0x7fc7bc0ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:00.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.179+0000 7fc7c189e700 1 -- 172.21.15.86:0/2028899926 shutdown_connections 2023-06-23T00:40:00.186 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.179+0000 7fc7c189e700 1 -- 172.21.15.86:0/2028899926 wait complete. 2023-06-23T00:40:00.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.180+0000 7fc7c189e700 1 Processor -- start 2023-06-23T00:40:00.188 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.180+0000 7fc7c189e700 1 -- start start 2023-06-23T00:40:00.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.181+0000 7fc7c189e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0b0b50 0x7fc7bc144d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.181+0000 7fc7c189e700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc1555b0 0x7fc7bc142a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.181+0000 7fc7c189e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7bc155980 0x7fc7bc13f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.181+0000 7fc7c189e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc148490 con 0x7fc7bc155980 2023-06-23T00:40:00.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.181+0000 7fc7c189e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc0cc660 con 0x7fc7bc0b0b50 2023-06-23T00:40:00.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.181+0000 7fc7c189e700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc147bf0 con 0x7fc7bc1555b0 2023-06-23T00:40:00.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.181+0000 7fc7baffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0b0b50 0x7fc7bc144d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.189 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.181+0000 7fc7ba7fc700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc1555b0 0x7fc7bc142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.181+0000 7fc7baffd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0b0b50 0x7fc7bc144d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38600/0 (socket says 172.21.15.86:38600) 2023-06-23T00:40:00.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.181+0000 7fc7baffd700 1 -- 172.21.15.86:0/3716555140 learned_addr learned my addr 172.21.15.86:0/3716555140 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-06-23T00:40:00.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.181+0000 7fc7bb7fe700 1 --2- 172.21.15.86:0/3716555140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7bc155980 0x7fc7bc13f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7ba7fc700 1 -- 172.21.15.86:0/3716555140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0b0b50 msgr2=0x7fc7bc144d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7ba7fc700 1 --2- 172.21.15.86:0/3716555140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0b0b50 0x7fc7bc144d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7ba7fc700 1 -- 172.21.15.86:0/3716555140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7bc155980 msgr2=0x7fc7bc13f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.190 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7ba7fc700 1 --2- 172.21.15.86:0/3716555140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7bc155980 0x7fc7bc13f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7ba7fc700 1 -- 172.21.15.86:0/3716555140 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b000a5d0 con 0x7fc7bc1555b0 2023-06-23T00:40:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7baffd700 1 --2- 172.21.15.86:0/3716555140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0b0b50 0x7fc7bc144d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7ba7fc700 1 --2- 172.21.15.86:0/3716555140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc1555b0 0x7fc7bc142a50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0017000 tx=0x7fc7b001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7a2ffd700 1 -- 172.21.15.86:0/3716555140 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b0019e70 con 0x7fc7bc1555b0 2023-06-23T00:40:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7a2ffd700 1 -- 172.21.15.86:0/3716555140 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b0021430 con 0x7fc7bc1555b0 2023-06-23T00:40:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7c189e700 1 -- 172.21.15.86:0/3716555140 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc147ed0 con 0x7fc7bc1555b0 2023-06-23T00:40:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7a2ffd700 1 -- 172.21.15.86:0/3716555140 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b00217d0 con 0x7fc7bc1555b0 2023-06-23T00:40:00.191 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7c189e700 1 -- 172.21.15.86:0/3716555140 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc1402b0 con 0x7fc7bc1555b0 2023-06-23T00:40:00.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7a2ffd700 1 -- 172.21.15.86:0/3716555140 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc7b002b440 con 0x7fc7bc1555b0 2023-06-23T00:40:00.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.182+0000 7fc7a2ffd700 1 -- 172.21.15.86:0/3716555140 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fc7b0021e40 con 0x7fc7bc1555b0 2023-06-23T00:40:00.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.183+0000 7fc7c189e700 1 -- 172.21.15.86:0/3716555140 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc04c1b0 con 0x7fc7bc1555b0 2023-06-23T00:40:00.192 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.185+0000 7fc7a2ffd700 1 -- 172.21.15.86:0/3716555140 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fc7b0026070 con 0x7fc7bc1555b0 2023-06-23T00:40:00.241 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:00.238+0000 7f84c915b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T00:40:00.248 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:00.245+0000 7fbcd7ef5180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T00:40:00.271 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.267+0000 7fc7c189e700 1 -- 172.21.15.86:0/3716555140 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7e40bfd5-0f9f-448c-8ccf-2f44bf82a622", "id": 6} v 0) v1 -- 0x7fc7bc1404a0 con 0x7fc7bc1555b0 2023-06-23T00:40:00.276 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.273+0000 7fc7a2ffd700 1 -- 172.21.15.86:0/3716555140 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7e40bfd5-0f9f-448c-8ccf-2f44bf82a622", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc7b000b040 con 0x7fc7bc1555b0 2023-06-23T00:40:00.277 INFO:teuthology.orchestra.run.smithi086.stdout:6 2023-06-23T00:40:00.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.274+0000 7fc7c189e700 1 -- 172.21.15.86:0/3716555140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc1555b0 msgr2=0x7fc7bc142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.274+0000 7fc7c189e700 1 --2- 172.21.15.86:0/3716555140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc1555b0 0x7fc7bc142a50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0017000 tx=0x7fc7b001a000 comp rx=0 tx=0).stop 2023-06-23T00:40:00.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.274+0000 7fc7c189e700 1 -- 172.21.15.86:0/3716555140 shutdown_connections 2023-06-23T00:40:00.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.274+0000 7fc7c189e700 1 --2- 172.21.15.86:0/3716555140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7bc155980 0x7fc7bc13f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.274+0000 7fc7c189e700 1 --2- 172.21.15.86:0/3716555140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0b0b50 0x7fc7bc144d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.274+0000 7fc7c189e700 1 --2- 172.21.15.86:0/3716555140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc1555b0 0x7fc7bc142a50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.278 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.274+0000 7fc7c189e700 1 -- 172.21.15.86:0/3716555140 >> 172.21.15.86:0/3716555140 conn(0x7fc7bc068fb0 msgr2=0x7fc7bc148de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:00.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.274+0000 7fc7c189e700 1 -- 172.21.15.86:0/3716555140 shutdown_connections 2023-06-23T00:40:00.279 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.274+0000 7fc7c189e700 1 -- 172.21.15.86:0/3716555140 wait complete. 2023-06-23T00:40:00.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a 7 2023-06-23T00:40:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.402+0000 7efcb5486700 1 Processor -- start 2023-06-23T00:40:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.402+0000 7efcb5486700 1 -- start start 2023-06-23T00:40:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efcb5486700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcb0157620 0x7efcb00acda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efcb5486700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb00ac910 con 0x7efcb0157250 2023-06-23T00:40:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efcb5486700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb00abd00 con 0x7efcb00ad2e0 2023-06-23T00:40:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efcb5486700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb00abfe0 con 0x7efcb0157620 2023-06-23T00:40:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efca7fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcb0157620 0x7efcb00acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efcaf7fe700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7efcb00ad2e0 0x7efcb00af480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40190/0 (socket says 172.21.15.86:40190) 2023-06-23T00:40:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efca7fff700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcb0157620 0x7efcb00acda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38604/0 (socket says 172.21.15.86:38604) 2023-06-23T00:40:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efcaf7fe700 1 -- 172.21.15.86:0/3912330122 learned_addr learned my addr 172.21.15.86:0/3912330122 (peer_addr_for_me v1:172.21.15.86:0/0) 2023-06-23T00:40:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efca7fff700 1 -- 172.21.15.86:0/3912330122 >> v1:172.21.15.86:6789/0 conn(0x7efcb00ad2e0 legacy=0x7efcb00af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efca7fff700 1 -- 172.21.15.86:0/3912330122 >> v1:172.21.15.64:6789/0 conn(0x7efcb0157250 legacy=0x7efcb006f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efca7fff700 1 -- 172.21.15.86:0/3912330122 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb00ac2c0 con 0x7efcb0157620 2023-06-23T00:40:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efca7fff700 1 --2- 172.21.15.86:0/3912330122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcb0157620 0x7efcb00acda0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efc980098e0 tx=0x7efc980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ce55c619893ee28 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efcae7fc700 1 -- 172.21.15.86:0/3912330122 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98009e80 con 0x7efcb0157620 2023-06-23T00:40:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efcae7fc700 1 -- 172.21.15.86:0/3912330122 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9800b860 con 0x7efcb0157620 2023-06-23T00:40:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.403+0000 7efcae7fc700 1 -- 172.21.15.86:0/3912330122 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9800bbe0 con 0x7efcb0157620 2023-06-23T00:40:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.404+0000 7efcb5486700 1 -- 172.21.15.86:0/3912330122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcb0157620 msgr2=0x7efcb00acda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.404+0000 7efcb5486700 1 --2- 172.21.15.86:0/3912330122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcb0157620 0x7efcb00acda0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7efc980098e0 tx=0x7efc980091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.404+0000 7efcb5486700 1 -- 172.21.15.86:0/3912330122 shutdown_connections 2023-06-23T00:40:00.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.404+0000 7efcb5486700 1 --2- 172.21.15.86:0/3912330122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcb0157620 0x7efcb00acda0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.404+0000 7efcb5486700 1 -- 172.21.15.86:0/3912330122 >> 172.21.15.86:0/3912330122 conn(0x7efcb00cd180 msgr2=0x7efcb00c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:00.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.404+0000 7efcb5486700 1 -- 172.21.15.86:0/3912330122 shutdown_connections 2023-06-23T00:40:00.410 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.404+0000 7efcb5486700 1 -- 172.21.15.86:0/3912330122 wait complete. 2023-06-23T00:40:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.405+0000 7efcb5486700 1 Processor -- start 2023-06-23T00:40:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.405+0000 7efcb5486700 1 -- start start 2023-06-23T00:40:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efcb5486700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcb00ad2e0 0x7efcb01503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efcb5486700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcb0157250 0x7efcb014bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efcb5486700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0157620 0x7efcb00c36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efcb5486700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb014e100 con 0x7efcb0157620 2023-06-23T00:40:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efcb5486700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb014ec30 con 0x7efcb0157250 2023-06-23T00:40:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efcb5486700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb0150900 con 0x7efcb00ad2e0 2023-06-23T00:40:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca7fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcb0157250 0x7efcb014bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efcaeffd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcb00ad2e0 0x7efcb01503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca7fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcb0157250 0x7efcb014bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38612/0 (socket says 172.21.15.86:38612) 2023-06-23T00:40:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca7fff700 1 -- 172.21.15.86:0/3841137737 learned_addr learned my addr 172.21.15.86:0/3841137737 (peer_addr_for_me v2:172.21.15.86:0/0) 2023-06-23T00:40:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efcaf7fe700 1 --2- 172.21.15.86:0/3841137737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0157620 0x7efcb00c36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca7fff700 1 -- 172.21.15.86:0/3841137737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcb00ad2e0 msgr2=0x7efcb01503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca7fff700 1 --2- 172.21.15.86:0/3841137737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcb00ad2e0 0x7efcb01503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca7fff700 1 -- 172.21.15.86:0/3841137737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0157620 msgr2=0x7efcb00c36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca7fff700 1 --2- 172.21.15.86:0/3841137737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0157620 0x7efcb00c36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca7fff700 1 -- 172.21.15.86:0/3841137737 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc98009420 con 0x7efcb0157250 2023-06-23T00:40:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efcaeffd700 1 --2- 172.21.15.86:0/3841137737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcb00ad2e0 0x7efcb01503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:00.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca7fff700 1 --2- 172.21.15.86:0/3841137737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcb0157250 0x7efcb014bce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efc9800b3c0 tx=0x7efc98017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:00.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca6ffd700 1 -- 172.21.15.86:0/3841137737 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9800b9d0 con 0x7efcb0157250 2023-06-23T00:40:00.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca6ffd700 1 -- 172.21.15.86:0/3841137737 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9800bb30 con 0x7efcb0157250 2023-06-23T00:40:00.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efcb5486700 1 -- 172.21.15.86:0/3841137737 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb014e300 con 0x7efcb0157250 2023-06-23T00:40:00.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efca6ffd700 1 -- 172.21.15.86:0/3841137737 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98025630 con 0x7efcb0157250 2023-06-23T00:40:00.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.406+0000 7efcb5486700 1 -- 172.21.15.86:0/3841137737 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb014e8c0 con 0x7efcb0157250 2023-06-23T00:40:00.415 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.407+0000 7efca6ffd700 1 -- 172.21.15.86:0/3841137737 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7efc98023070 con 0x7efcb0157250 2023-06-23T00:40:00.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.407+0000 7efca6ffd700 1 -- 172.21.15.86:0/3841137737 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7efc9802fb00 con 0x7efcb0157250 2023-06-23T00:40:00.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.407+0000 7efcb5486700 1 -- 172.21.15.86:0/3841137737 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb004c1b0 con 0x7efcb0157250 2023-06-23T00:40:00.416 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.409+0000 7efca6ffd700 1 -- 172.21.15.86:0/3841137737 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7efc98034060 con 0x7efcb0157250 2023-06-23T00:40:00.495 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.491+0000 7efcb5486700 1 -- 172.21.15.86:0/3841137737 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a", "id": 7} v 0) v1 -- 0x7efcb0068920 con 0x7efcb0157250 2023-06-23T00:40:00.502 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.499+0000 7efca6ffd700 1 -- 172.21.15.86:0/3841137737 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7efc9802a070 con 0x7efcb0157250 2023-06-23T00:40:00.503 INFO:teuthology.orchestra.run.smithi086.stdout:7 2023-06-23T00:40:00.503 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.500+0000 7efcb5486700 1 -- 172.21.15.86:0/3841137737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcb0157250 msgr2=0x7efcb014bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.504 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.500+0000 7efcb5486700 1 --2- 172.21.15.86:0/3841137737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcb0157250 0x7efcb014bce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efc9800b3c0 tx=0x7efc98017000 comp rx=0 tx=0).stop 2023-06-23T00:40:00.504 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.500+0000 7efcb5486700 1 -- 172.21.15.86:0/3841137737 shutdown_connections 2023-06-23T00:40:00.504 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.500+0000 7efcb5486700 1 --2- 172.21.15.86:0/3841137737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efcb0157620 0x7efcb00c36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.504 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.500+0000 7efcb5486700 1 --2- 172.21.15.86:0/3841137737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcb00ad2e0 0x7efcb01503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.504 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.500+0000 7efcb5486700 1 --2- 172.21.15.86:0/3841137737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcb0157250 0x7efcb014bce0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.504 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.500+0000 7efcb5486700 1 -- 172.21.15.86:0/3841137737 >> 172.21.15.86:0/3841137737 conn(0x7efcb00cd180 msgr2=0x7efcb014cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:00.505 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.500+0000 7efcb5486700 1 -- 172.21.15.86:0/3841137737 shutdown_connections 2023-06-23T00:40:00.505 INFO:teuthology.orchestra.run.smithi086.stderr:2023-06-23T00:40:00.500+0000 7efcb5486700 1 -- 172.21.15.86:0/3841137737 wait complete. 2023-06-23T00:40:00.515 INFO:tasks.ceph.osd.0:Restarting daemon 2023-06-23T00:40:00.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-06-23T00:40:00.518 INFO:tasks.ceph.osd.0:Started 2023-06-23T00:40:00.518 INFO:tasks.ceph.osd.1:Restarting daemon 2023-06-23T00:40:00.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-06-23T00:40:00.520 INFO:tasks.ceph.osd.1:Started 2023-06-23T00:40:00.520 INFO:tasks.ceph.osd.2:Restarting daemon 2023-06-23T00:40:00.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-06-23T00:40:00.522 INFO:tasks.ceph.osd.2:Started 2023-06-23T00:40:00.523 INFO:tasks.ceph.osd.3:Restarting daemon 2023-06-23T00:40:00.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-06-23T00:40:00.525 INFO:tasks.ceph.osd.3:Started 2023-06-23T00:40:00.525 INFO:tasks.ceph.osd.4:Restarting daemon 2023-06-23T00:40:00.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-06-23T00:40:00.527 INFO:tasks.ceph.osd.4:Started 2023-06-23T00:40:00.527 INFO:tasks.ceph.osd.5:Restarting daemon 2023-06-23T00:40:00.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-06-23T00:40:00.530 INFO:tasks.ceph.osd.5:Started 2023-06-23T00:40:00.530 INFO:tasks.ceph.osd.6:Restarting daemon 2023-06-23T00:40:00.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-06-23T00:40:00.532 INFO:tasks.ceph.osd.6:Started 2023-06-23T00:40:00.532 INFO:tasks.ceph.osd.7:Restarting daemon 2023-06-23T00:40:00.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-06-23T00:40:00.534 INFO:tasks.ceph.osd.7:Started 2023-06-23T00:40:00.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T00:40:00.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.680+0000 7f9a6bf62700 1 Processor -- start 2023-06-23T00:40:00.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.680+0000 7f9a6bf62700 1 -- start start 2023-06-23T00:40:00.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.681+0000 7f9a6bf62700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a641570c0 0x7f9a6415a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.681+0000 7f9a6bf62700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a640d08c0 0x7f9a640cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.681+0000 7f9a6bf62700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6415aaa0 con 0x7f9a640d04f0 2023-06-23T00:40:00.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.681+0000 7f9a6bf62700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a640ce5c0 con 0x7f9a641570c0 2023-06-23T00:40:00.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.681+0000 7f9a6bf62700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640ccf70 con 0x7f9a640d08c0 2023-06-23T00:40:00.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.681+0000 7f9a69cfe700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f9a640d04f0 0x7f9a641569b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36060/0 (socket says 172.21.15.64:36060) 2023-06-23T00:40:00.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.681+0000 7f9a69cfe700 1 -- 172.21.15.64:0/4195029668 learned_addr learned my addr 172.21.15.64:0/4195029668 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:00.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.681+0000 7f9a6a4ff700 1 --2- 172.21.15.64:0/4195029668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a640d08c0 0x7f9a640cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.681+0000 7f9a694fd700 1 --2- 172.21.15.64:0/4195029668 >> v2:172.21.15.86:3300/0 conn(0x7f9a641570c0 0x7f9a6415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.682+0000 7f9a68cfc700 1 -- 172.21.15.64:0/4195029668 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001541850 0 0) 0x7f9a6415aaa0 con 0x7f9a640d04f0 2023-06-23T00:40:00.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.682+0000 7f9a68cfc700 1 -- 172.21.15.64:0/4195029668 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a4c003570 con 0x7f9a640d04f0 2023-06-23T00:40:00.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.682+0000 7f9a6a4ff700 1 -- 172.21.15.64:0/4195029668 >> v2:172.21.15.86:3300/0 conn(0x7f9a641570c0 msgr2=0x7f9a6415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:00.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.682+0000 7f9a6a4ff700 1 --2- 172.21.15.64:0/4195029668 >> v2:172.21.15.86:3300/0 conn(0x7f9a641570c0 0x7f9a6415a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.682+0000 7f9a6a4ff700 1 -- 172.21.15.64:0/4195029668 >> v1:172.21.15.64:6789/0 conn(0x7f9a640d04f0 legacy=0x7f9a641569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.682+0000 7f9a6a4ff700 1 -- 172.21.15.64:0/4195029668 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a640cd1d0 con 0x7f9a640d08c0 2023-06-23T00:40:00.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.682+0000 7f9a694fd700 1 --2- 172.21.15.64:0/4195029668 >> v2:172.21.15.86:3300/0 conn(0x7f9a641570c0 0x7f9a6415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:00.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.682+0000 7f9a6a4ff700 1 --2- 172.21.15.64:0/4195029668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a640d08c0 0x7f9a640cca30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9a54009e60 tx=0x7f9a54009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95ebeeba915ed6bc server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:00.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.682+0000 7f9a68cfc700 1 -- 172.21.15.64:0/4195029668 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400e070 con 0x7f9a640d08c0 2023-06-23T00:40:00.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.682+0000 7f9a68cfc700 1 -- 172.21.15.64:0/4195029668 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5400bab0 con 0x7f9a640d08c0 2023-06-23T00:40:00.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.682+0000 7f9a68cfc700 1 -- 172.21.15.64:0/4195029668 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400be70 con 0x7f9a640d08c0 2023-06-23T00:40:00.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.683+0000 7f9a6bf62700 1 -- 172.21.15.64:0/4195029668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a640d08c0 msgr2=0x7f9a640cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.683+0000 7f9a6bf62700 1 --2- 172.21.15.64:0/4195029668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a640d08c0 0x7f9a640cca30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9a54009e60 tx=0x7f9a54009780 comp rx=0 tx=0).stop 2023-06-23T00:40:00.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.683+0000 7f9a6bf62700 1 -- 172.21.15.64:0/4195029668 shutdown_connections 2023-06-23T00:40:00.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.683+0000 7f9a6bf62700 1 --2- 172.21.15.64:0/4195029668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a640d08c0 0x7f9a640cca30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.683+0000 7f9a6bf62700 1 --2- 172.21.15.64:0/4195029668 >> v2:172.21.15.86:3300/0 conn(0x7f9a641570c0 0x7f9a6415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.683+0000 7f9a6bf62700 1 -- 172.21.15.64:0/4195029668 >> 172.21.15.64:0/4195029668 conn(0x7f9a6406a060 msgr2=0x7f9a6406a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:00.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.683+0000 7f9a6bf62700 1 -- 172.21.15.64:0/4195029668 shutdown_connections 2023-06-23T00:40:00.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.683+0000 7f9a6bf62700 1 -- 172.21.15.64:0/4195029668 wait complete. 2023-06-23T00:40:00.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.684+0000 7f9a6bf62700 1 Processor -- start 2023-06-23T00:40:00.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.684+0000 7f9a6bf62700 1 -- start start 2023-06-23T00:40:00.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a6bf62700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a640d04f0 0x7f9a64147cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a6bf62700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a640d08c0 0x7f9a640b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a6bf62700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a641570c0 0x7f9a641546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:00.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a6bf62700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640b4230 con 0x7f9a640d08c0 2023-06-23T00:40:00.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a6bf62700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640c68a0 con 0x7f9a641570c0 2023-06-23T00:40:00.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a6bf62700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640b3f50 con 0x7f9a640d04f0 2023-06-23T00:40:00.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a694fd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a640d08c0 0x7f9a640b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a694fd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a640d08c0 0x7f9a640b57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55416/0 (socket says 172.21.15.64:55416) 2023-06-23T00:40:00.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a694fd700 1 -- 172.21.15.64:0/476762453 learned_addr learned my addr 172.21.15.64:0/476762453 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:00.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a69cfe700 1 --2- 172.21.15.64:0/476762453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a640d04f0 0x7f9a64147cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a6a4ff700 1 --2- 172.21.15.64:0/476762453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a641570c0 0x7f9a641546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:00.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a694fd700 1 -- 172.21.15.64:0/476762453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a640d04f0 msgr2=0x7f9a64147cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a694fd700 1 --2- 172.21.15.64:0/476762453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a640d04f0 0x7f9a64147cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a694fd700 1 -- 172.21.15.64:0/476762453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a641570c0 msgr2=0x7f9a641546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a694fd700 1 --2- 172.21.15.64:0/476762453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a641570c0 0x7f9a641546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a694fd700 1 -- 172.21.15.64:0/476762453 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a58009420 con 0x7f9a640d08c0 2023-06-23T00:40:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a694fd700 1 --2- 172.21.15.64:0/476762453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a640d08c0 0x7f9a640b57c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800ec80 tx=0x7f9a580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a61ffb700 1 -- 172.21.15.64:0/476762453 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5800c850 con 0x7f9a640d08c0 2023-06-23T00:40:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a61ffb700 1 -- 172.21.15.64:0/476762453 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5800c9b0 con 0x7f9a640d08c0 2023-06-23T00:40:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a6bf62700 1 -- 172.21.15.64:0/476762453 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a54009a20 con 0x7f9a640d08c0 2023-06-23T00:40:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.685+0000 7f9a61ffb700 1 -- 172.21.15.64:0/476762453 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5800cd30 con 0x7f9a640d08c0 2023-06-23T00:40:00.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.686+0000 7f9a6bf62700 1 -- 172.21.15.64:0/476762453 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a640c6530 con 0x7f9a640d08c0 2023-06-23T00:40:00.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.686+0000 7f9a61ffb700 1 -- 172.21.15.64:0/476762453 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9a58005c80 con 0x7f9a640d08c0 2023-06-23T00:40:00.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.686+0000 7f9a61ffb700 1 -- 172.21.15.64:0/476762453 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f9a58006700 con 0x7f9a640d08c0 2023-06-23T00:40:00.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.686+0000 7f9a6bf62700 1 -- 172.21.15.64:0/476762453 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a640c6a30 con 0x7f9a640d08c0 2023-06-23T00:40:00.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.689+0000 7f9a61ffb700 1 -- 172.21.15.64:0/476762453 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f9a58021060 con 0x7f9a640d08c0 2023-06-23T00:40:00.764 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.760+0000 7f9a6bf62700 1 -- 172.21.15.64:0/476762453 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a6404c1b0 con 0x7f9a640d08c0 2023-06-23T00:40:00.764 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.761+0000 7f9a61ffb700 1 -- 172.21.15.64:0/476762453 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f9a58017070 con 0x7f9a640d08c0 2023-06-23T00:40:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:00.765 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":10,"fsid":"72efde62-4c91-4277-87ee-356e50bd4cf5","created":"2023-06-23T00:39:58.078232+0000","modified":"2023-06-23T00:40:00.493656+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T00:40:00.493656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f0698c9d-c67a-42ac-8d36-838836c514d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5ece869-9da7-40e8-8f1d-bbaf67bb2187","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"21067fac-663b-4ecf-ae67-bf235161638e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0e987a73-8b36-4f26-94fd-f0135d6062c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7e40bfd5-0f9f-448c-8ccf-2f44bf82a622","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T00:40:00.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.762+0000 7f9a6bf62700 1 -- 172.21.15.64:0/476762453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a640d08c0 msgr2=0x7f9a640b57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:00.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.762+0000 7f9a6bf62700 1 --2- 172.21.15.64:0/476762453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a640d08c0 0x7f9a640b57c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800ec80 tx=0x7f9a580091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.762+0000 7f9a6bf62700 1 -- 172.21.15.64:0/476762453 shutdown_connections 2023-06-23T00:40:00.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.762+0000 7f9a6bf62700 1 --2- 172.21.15.64:0/476762453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a641570c0 0x7f9a641546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.762+0000 7f9a6bf62700 1 --2- 172.21.15.64:0/476762453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a640d04f0 0x7f9a64147cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.762+0000 7f9a6bf62700 1 --2- 172.21.15.64:0/476762453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9a640d08c0 0x7f9a640b57c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:00.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.762+0000 7f9a6bf62700 1 -- 172.21.15.64:0/476762453 >> 172.21.15.64:0/476762453 conn(0x7f9a6406a060 msgr2=0x7f9a641574d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:00.767 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.762+0000 7f9a6bf62700 1 -- 172.21.15.64:0/476762453 shutdown_connections 2023-06-23T00:40:00.767 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:00.762+0000 7f9a6bf62700 1 -- 172.21.15.64:0/476762453 wait complete. 2023-06-23T00:40:00.777 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-06-23T00:40:00.778 INFO:tasks.ceph:Waiting for OSDs to come up 2023-06-23T00:40:00.785 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:00.782+0000 7f84c915b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T00:40:00.815 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:00.812+0000 7fbcd7ef5180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T00:40:00.862 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:00.859+0000 7f84c915b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T00:40:00.896 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:00.892+0000 7fbcd7ef5180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T00:40:00.941 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:00.938+0000 7f84c915b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T00:40:00.976 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:00.973+0000 7fbcd7ef5180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T00:40:01.124 INFO:tasks.ceph.osd.4.smithi086.stderr:2023-06-23T00:40:01.120+0000 7fdadb5fb540 -1 Falling back to public interface 2023-06-23T00:40:01.130 INFO:tasks.ceph.osd.7.smithi086.stderr:2023-06-23T00:40:01.127+0000 7fc4fc5d9540 -1 Falling back to public interface 2023-06-23T00:40:01.131 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-06-23T00:40:01.127+0000 7f89a8296540 -1 Falling back to public interface 2023-06-23T00:40:01.146 INFO:tasks.ceph.osd.5.smithi086.stderr:2023-06-23T00:40:01.143+0000 7fa925fac540 -1 Falling back to public interface 2023-06-23T00:40:01.149 INFO:tasks.ceph.osd.6.smithi086.stderr:2023-06-23T00:40:01.146+0000 7f39aaaff540 -1 Falling back to public interface 2023-06-23T00:40:01.162 INFO:tasks.ceph.osd.3.smithi064.stderr:2023-06-23T00:40:01.158+0000 7ff028268540 -1 Falling back to public interface 2023-06-23T00:40:01.166 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-06-23T00:40:01.163+0000 7fed89ca3540 -1 Falling back to public interface 2023-06-23T00:40:01.177 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-06-23T00:40:01.173+0000 7fcc9c12e540 -1 Falling back to public interface 2023-06-23T00:40:01.524 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:01.521+0000 7f84c915b180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T00:40:01.584 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T00:40:01.589 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:01.585+0000 7fbcd7ef5180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T00:40:01.663 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:01.660+0000 7f84c915b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T00:40:01.695 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.691+0000 7fa431bea700 1 Processor -- start 2023-06-23T00:40:01.695 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.692+0000 7fa431bea700 1 -- start start 2023-06-23T00:40:01.697 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.692+0000 7fa431bea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c153c40 0x7fa42c0b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:01.697 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.692+0000 7fa431bea700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fa42c0af110 0x7fa42c0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:01.697 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.692+0000 7fa431bea700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa42c0b2a30 con 0x7fa42c069a80 2023-06-23T00:40:01.697 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.692+0000 7fa431bea700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa42c0b2d10 con 0x7fa42c0af110 2023-06-23T00:40:01.697 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.692+0000 7fa431bea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c068d90 con 0x7fa42c153c40 2023-06-23T00:40:01.698 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42bfff700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fa42c0af110 0x7fa42c0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:01.698 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42b7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c153c40 0x7fa42c0b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:01.698 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa422ffd700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa42c069a80 0x7fa42c0aea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:36454/0 (socket says 172.21.15.64:36454) 2023-06-23T00:40:01.698 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42bfff700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fa42c0af110 0x7fa42c0afdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55432/0 (socket says 172.21.15.64:55432) 2023-06-23T00:40:01.698 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa422ffd700 1 -- 172.21.15.64:0/1894983913 learned_addr learned my addr 172.21.15.64:0/1894983913 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:01.698 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42b7fe700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c153c40 0x7fa42c0b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.64:44836/0 (socket says 172.21.15.64:44836) 2023-06-23T00:40:01.698 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42b7fe700 1 -- 172.21.15.64:0/1894983913 >> v2:172.21.15.64:3300/0 conn(0x7fa42c0af110 msgr2=0x7fa42c0afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:01.699 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42b7fe700 1 --2- 172.21.15.64:0/1894983913 >> v2:172.21.15.64:3300/0 conn(0x7fa42c0af110 0x7fa42c0afdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:01.699 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42affd700 1 -- 172.21.15.64:0/1894983913 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331066104 0 0) 0x7fa42c0b2a30 con 0x7fa42c069a80 2023-06-23T00:40:01.699 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42b7fe700 1 -- 172.21.15.64:0/1894983913 >> v1:172.21.15.86:6789/0 conn(0x7fa42c069a80 legacy=0x7fa42c0aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:01.699 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42b7fe700 1 -- 172.21.15.64:0/1894983913 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c068fa0 con 0x7fa42c153c40 2023-06-23T00:40:01.699 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42b7fe700 1 --2- 172.21.15.64:0/1894983913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c153c40 0x7fa42c0b2020 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0098e0 tx=0x7fa41c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c93861470416d26a server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:01.699 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42affd700 1 -- 172.21.15.64:0/1894983913 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c00b640 con 0x7fa42c153c40 2023-06-23T00:40:01.699 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42affd700 1 -- 172.21.15.64:0/1894983913 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa41c00b7a0 con 0x7fa42c153c40 2023-06-23T00:40:01.699 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.693+0000 7fa42affd700 1 -- 172.21.15.64:0/1894983913 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c00bb40 con 0x7fa42c153c40 2023-06-23T00:40:01.702 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.694+0000 7fa431bea700 1 -- 172.21.15.64:0/1894983913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c153c40 msgr2=0x7fa42c0b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:01.702 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.694+0000 7fa431bea700 1 --2- 172.21.15.64:0/1894983913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c153c40 0x7fa42c0b2020 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0098e0 tx=0x7fa41c0091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:01.702 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.694+0000 7fa431bea700 1 -- 172.21.15.64:0/1894983913 shutdown_connections 2023-06-23T00:40:01.702 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.694+0000 7fa431bea700 1 --2- 172.21.15.64:0/1894983913 >> v2:172.21.15.64:3300/0 conn(0x7fa42c0af110 0x7fa42c0afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:01.702 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.694+0000 7fa431bea700 1 --2- 172.21.15.64:0/1894983913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c153c40 0x7fa42c0b2020 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:01.702 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.694+0000 7fa431bea700 1 -- 172.21.15.64:0/1894983913 >> 172.21.15.64:0/1894983913 conn(0x7fa42c06b280 msgr2=0x7fa42c06b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:01.703 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.694+0000 7fa431bea700 1 -- 172.21.15.64:0/1894983913 shutdown_connections 2023-06-23T00:40:01.703 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.694+0000 7fa431bea700 1 -- 172.21.15.64:0/1894983913 wait complete. 2023-06-23T00:40:01.703 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.695+0000 7fa431bea700 1 Processor -- start 2023-06-23T00:40:01.703 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.695+0000 7fa431bea700 1 -- start start 2023-06-23T00:40:01.703 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa431bea700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c069a80 0x7fa42c166d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:01.703 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa431bea700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa42c0af110 0x7fa42c167f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:01.703 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa431bea700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa42c153c40 0x7fa42c0cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:01.704 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa431bea700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c0cbf90 con 0x7fa42c0af110 2023-06-23T00:40:01.704 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa431bea700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c1506d0 con 0x7fa42c153c40 2023-06-23T00:40:01.704 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa431bea700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa42c0cc5b0 con 0x7fa42c069a80 2023-06-23T00:40:01.704 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa422ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa42c0af110 0x7fa42c167f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:01.704 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa422ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa42c0af110 0x7fa42c167f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:55434/0 (socket says 172.21.15.64:55434) 2023-06-23T00:40:01.704 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa422ffd700 1 -- 172.21.15.64:0/3490861737 learned_addr learned my addr 172.21.15.64:0/3490861737 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:01.704 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa42b7fe700 1 --2- 172.21.15.64:0/3490861737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c069a80 0x7fa42c166d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:01.704 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa42bfff700 1 --2- 172.21.15.64:0/3490861737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa42c153c40 0x7fa42c0cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:01.705 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa422ffd700 1 -- 172.21.15.64:0/3490861737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c069a80 msgr2=0x7fa42c166d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:01.705 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa422ffd700 1 --2- 172.21.15.64:0/3490861737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c069a80 0x7fa42c166d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:01.705 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa422ffd700 1 -- 172.21.15.64:0/3490861737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa42c153c40 msgr2=0x7fa42c0cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:01.705 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa422ffd700 1 --2- 172.21.15.64:0/3490861737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa42c153c40 0x7fa42c0cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:01.705 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.696+0000 7fa422ffd700 1 -- 172.21.15.64:0/3490861737 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c009420 con 0x7fa42c0af110 2023-06-23T00:40:01.705 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.697+0000 7fa422ffd700 1 --2- 172.21.15.64:0/3490861737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa42c0af110 0x7fa42c167f30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa41400c810 tx=0x7fa414007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:01.705 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.697+0000 7fa423fff700 1 -- 172.21.15.64:0/3490861737 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414005c00 con 0x7fa42c0af110 2023-06-23T00:40:01.706 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.697+0000 7fa423fff700 1 -- 172.21.15.64:0/3490861737 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa414005d60 con 0x7fa42c0af110 2023-06-23T00:40:01.706 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.697+0000 7fa431bea700 1 -- 172.21.15.64:0/3490861737 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42c167350 con 0x7fa42c0af110 2023-06-23T00:40:01.706 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.697+0000 7fa423fff700 1 -- 172.21.15.64:0/3490861737 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414007300 con 0x7fa42c0af110 2023-06-23T00:40:01.706 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.697+0000 7fa431bea700 1 -- 172.21.15.64:0/3490861737 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42c167910 con 0x7fa42c0af110 2023-06-23T00:40:01.706 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.697+0000 7fa423fff700 1 -- 172.21.15.64:0/3490861737 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa41401c6c0 con 0x7fa42c0af110 2023-06-23T00:40:01.706 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.697+0000 7fa423fff700 1 -- 172.21.15.64:0/3490861737 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fa4140076f0 con 0x7fa42c0af110 2023-06-23T00:40:01.706 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.698+0000 7fa431bea700 1 -- 172.21.15.64:0/3490861737 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa42c0cc7b0 con 0x7fa42c0af110 2023-06-23T00:40:01.707 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.700+0000 7fa423fff700 1 -- 172.21.15.64:0/3490861737 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fa42c0cc7b0 con 0x7fa42c0af110 2023-06-23T00:40:01.736 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:01.733+0000 7fbcd7ef5180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T00:40:01.757 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:01.754+0000 7f84c915b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T00:40:01.774 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.770+0000 7fa431bea700 1 -- 172.21.15.64:0/3490861737 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa42c15ae00 con 0x7fa42c0af110 2023-06-23T00:40:01.774 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.771+0000 7fa423fff700 1 -- 172.21.15.64:0/3490861737 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fa414017070 con 0x7fa42c0af110 2023-06-23T00:40:01.775 INFO:teuthology.misc.health.smithi064.stdout: 2023-06-23T00:40:01.775 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":10,"fsid":"72efde62-4c91-4277-87ee-356e50bd4cf5","created":"2023-06-23T00:39:58.078232+0000","modified":"2023-06-23T00:40:00.493656+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T00:40:00.493656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f0698c9d-c67a-42ac-8d36-838836c514d6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5ece869-9da7-40e8-8f1d-bbaf67bb2187","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"21067fac-663b-4ecf-ae67-bf235161638e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0e987a73-8b36-4f26-94fd-f0135d6062c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7e40bfd5-0f9f-448c-8ccf-2f44bf82a622","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T00:40:01.776 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.772+0000 7fa431bea700 1 -- 172.21.15.64:0/3490861737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa42c0af110 msgr2=0x7fa42c167f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:01.776 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.772+0000 7fa431bea700 1 --2- 172.21.15.64:0/3490861737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa42c0af110 0x7fa42c167f30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa41400c810 tx=0x7fa414007db0 comp rx=0 tx=0).stop 2023-06-23T00:40:01.776 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.772+0000 7fa431bea700 1 -- 172.21.15.64:0/3490861737 shutdown_connections 2023-06-23T00:40:01.776 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.772+0000 7fa431bea700 1 --2- 172.21.15.64:0/3490861737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa42c153c40 0x7fa42c0cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:01.776 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.772+0000 7fa431bea700 1 --2- 172.21.15.64:0/3490861737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa42c069a80 0x7fa42c166d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:01.776 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.772+0000 7fa431bea700 1 --2- 172.21.15.64:0/3490861737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa42c0af110 0x7fa42c167f30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:01.777 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.772+0000 7fa431bea700 1 -- 172.21.15.64:0/3490861737 >> 172.21.15.64:0/3490861737 conn(0x7fa42c06b280 msgr2=0x7fa42c0ac950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:01.777 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.772+0000 7fa431bea700 1 -- 172.21.15.64:0/3490861737 shutdown_connections 2023-06-23T00:40:01.777 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:01.772+0000 7fa431bea700 1 -- 172.21.15.64:0/3490861737 wait complete. 2023-06-23T00:40:01.786 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-06-23T00:40:01.839 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:01.835+0000 7fbcd7ef5180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T00:40:02.004 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:02.001+0000 7f84c915b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T00:40:02.081 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:02.078+0000 7f84c915b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T00:40:02.096 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:02.093+0000 7fbcd7ef5180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T00:40:02.177 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:02.173+0000 7fbcd7ef5180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T00:40:02.897 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:02.894+0000 7f84c915b180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T00:40:02.969 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:02.966+0000 7f84c915b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T00:40:03.038 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:03.034+0000 7fbcd7ef5180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T00:40:03.043 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:03.040+0000 7f84c915b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T00:40:03.114 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:03.111+0000 7fbcd7ef5180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T00:40:03.177 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:03.173+0000 7f84c915b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T00:40:03.191 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:03.188+0000 7fbcd7ef5180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T00:40:03.260 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:03.257+0000 7f84c915b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T00:40:03.330 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:03.327+0000 7fbcd7ef5180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T00:40:03.413 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:03.410+0000 7f84c915b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T00:40:03.418 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:03.415+0000 7fbcd7ef5180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T00:40:03.490 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:03.486+0000 7f84c915b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T00:40:03.564 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:03.561+0000 7f84c915b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T00:40:03.575 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:03.572+0000 7fbcd7ef5180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T00:40:03.640 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:03.637+0000 7f84c915b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T00:40:03.663 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:03.659+0000 7fbcd7ef5180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T00:40:03.746 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:03.743+0000 7fbcd7ef5180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T00:40:03.824 INFO:tasks.ceph.mgr.y.smithi064.stderr:2023-06-23T00:40:03.821+0000 7fbcd7ef5180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T00:40:04.363 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-06-23T00:40:04.360+0000 7f89a8296540 -1 osd.0 0 log_to_monitors true 2023-06-23T00:40:04.418 INFO:tasks.ceph.osd.4.smithi086.stderr:2023-06-23T00:40:04.415+0000 7fdadb5fb540 -1 osd.4 0 log_to_monitors true 2023-06-23T00:40:04.661 INFO:tasks.ceph.osd.7.smithi086.stderr:2023-06-23T00:40:04.658+0000 7fc4fc5d9540 -1 osd.7 0 log_to_monitors true 2023-06-23T00:40:04.667 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:04.663+0000 7f848255f700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-06-23T00:40:04.667 INFO:tasks.ceph.mgr.x.smithi086.stderr:2023-06-23T00:40:04.664+0000 7f848255f700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-06-23T00:40:04.674 INFO:tasks.ceph.osd.3.smithi064.stderr:2023-06-23T00:40:04.671+0000 7ff028268540 -1 osd.3 0 log_to_monitors true 2023-06-23T00:40:04.691 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-06-23T00:40:04.688+0000 7fed89ca3540 -1 osd.2 0 log_to_monitors true 2023-06-23T00:40:04.694 INFO:tasks.ceph.osd.6.smithi086.stderr:2023-06-23T00:40:04.691+0000 7f39aaaff540 -1 osd.6 0 log_to_monitors true 2023-06-23T00:40:04.705 INFO:tasks.ceph.osd.5.smithi086.stderr:2023-06-23T00:40:04.702+0000 7fa925fac540 -1 osd.5 0 log_to_monitors true 2023-06-23T00:40:04.731 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-06-23T00:40:04.728+0000 7fcc9c12e540 -1 osd.1 0 log_to_monitors true 2023-06-23T00:40:07.880 INFO:tasks.ceph.osd.7.smithi086.stderr:2023-06-23T00:40:07.876+0000 7fc4f49fe700 -1 osd.7 0 waiting for initial osdmap 2023-06-23T00:40:07.884 INFO:tasks.ceph.osd.7.smithi086.stderr:2023-06-23T00:40:07.880+0000 7fc4ebff2700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T00:40:07.886 INFO:tasks.ceph.osd.5.smithi086.stderr:2023-06-23T00:40:07.882+0000 7fa91cbce700 -1 osd.5 0 waiting for initial osdmap 2023-06-23T00:40:07.889 INFO:tasks.ceph.osd.5.smithi086.stderr:2023-06-23T00:40:07.886+0000 7fa9159c5700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T00:40:07.898 INFO:tasks.ceph.osd.6.smithi086.stderr:2023-06-23T00:40:07.894+0000 7f39a1721700 -1 osd.6 0 waiting for initial osdmap 2023-06-23T00:40:07.901 INFO:tasks.ceph.osd.6.smithi086.stderr:2023-06-23T00:40:07.898+0000 7f399a518700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T00:40:07.907 INFO:tasks.ceph.osd.4.smithi086.stderr:2023-06-23T00:40:07.904+0000 7fdad3a20700 -1 osd.4 0 waiting for initial osdmap 2023-06-23T00:40:07.910 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-06-23T00:40:07.906+0000 7f899eeb8700 -1 osd.0 0 waiting for initial osdmap 2023-06-23T00:40:07.911 INFO:tasks.ceph.osd.4.smithi086.stderr:2023-06-23T00:40:07.908+0000 7fdacb014700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T00:40:07.918 INFO:tasks.ceph.osd.0.smithi064.stderr:2023-06-23T00:40:07.914+0000 7f8997caf700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T00:40:07.920 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-06-23T00:40:07.914+0000 7fcc92d50700 -1 osd.1 0 waiting for initial osdmap 2023-06-23T00:40:07.920 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-06-23T00:40:07.916+0000 7fed808c5700 -1 osd.2 0 waiting for initial osdmap 2023-06-23T00:40:07.921 INFO:tasks.ceph.osd.1.smithi064.stderr:2023-06-23T00:40:07.918+0000 7fcc8bb47700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T00:40:07.927 INFO:tasks.ceph.osd.2.smithi064.stderr:2023-06-23T00:40:07.924+0000 7fed796bc700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T00:40:07.948 INFO:tasks.ceph.osd.3.smithi064.stderr:2023-06-23T00:40:07.944+0000 7ff01ee8a700 -1 osd.3 0 waiting for initial osdmap 2023-06-23T00:40:07.955 INFO:tasks.ceph.osd.3.smithi064.stderr:2023-06-23T00:40:07.952+0000 7ff017c81700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T00:40:08.593 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T00:40:08.713 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.710+0000 7f2cdf26b700 1 Processor -- start 2023-06-23T00:40:08.714 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.710+0000 7f2cdf26b700 1 -- start start 2023-06-23T00:40:08.715 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.711+0000 7f2cdf26b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cd815d030 0x7f2cd8068730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:08.715 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.711+0000 7f2cdf26b700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd80682a0 con 0x7f2cd815d400 2023-06-23T00:40:08.716 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.711+0000 7f2cdf26b700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd80673f0 con 0x7f2cd80cf990 2023-06-23T00:40:08.716 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.711+0000 7f2cdf26b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd8067690 con 0x7f2cd815d030 2023-06-23T00:40:08.716 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.711+0000 7f2cdda68700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f2cd815d400 0x7f2cd80cf280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36596/0 (socket says 172.21.15.64:36596) 2023-06-23T00:40:08.716 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.711+0000 7f2cdda68700 1 -- 172.21.15.64:0/3454924769 learned_addr learned my addr 172.21.15.64:0/3454924769 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:08.716 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.711+0000 7f2cde269700 1 --2- 172.21.15.64:0/3454924769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cd815d030 0x7f2cd8068730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:08.716 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.711+0000 7f2cdd267700 1 -- 172.21.15.64:0/3454924769 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533527141 0 0) 0x7f2cd80682a0 con 0x7f2cd815d400 2023-06-23T00:40:08.716 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.711+0000 7f2cdd267700 1 -- 172.21.15.64:0/3454924769 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc4003570 con 0x7f2cd815d400 2023-06-23T00:40:08.717 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.711+0000 7f2cdd267700 1 -- 172.21.15.64:0/3454924769 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536508742 0 0) 0x7f2cd80673f0 con 0x7f2cd80cf990 2023-06-23T00:40:08.717 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.711+0000 7f2cdd267700 1 -- 172.21.15.64:0/3454924769 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd80682a0 con 0x7f2cd80cf990 2023-06-23T00:40:08.717 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.712+0000 7f2cde269700 1 -- 172.21.15.64:0/3454924769 >> v1:172.21.15.86:6789/0 conn(0x7f2cd80cf990 legacy=0x7f2cd8156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:08.717 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.712+0000 7f2cde269700 1 -- 172.21.15.64:0/3454924769 >> v1:172.21.15.64:6789/0 conn(0x7f2cd815d400 legacy=0x7f2cd80cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:08.717 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.712+0000 7f2cdd267700 1 -- 172.21.15.64:0/3454924769 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 713093870 0 0) 0x7f2cc4003570 con 0x7f2cd815d400 2023-06-23T00:40:08.717 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.712+0000 7f2cde269700 1 -- 172.21.15.64:0/3454924769 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd8067970 con 0x7f2cd815d030 2023-06-23T00:40:08.717 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.712+0000 7f2cde269700 1 --2- 172.21.15.64:0/3454924769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cd815d030 0x7f2cd8068730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0009f10 tx=0x7f2cd0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=384b7db7e7283c3b server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:08.718 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.712+0000 7f2cdd267700 1 -- 172.21.15.64:0/3454924769 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd000e070 con 0x7f2cd815d030 2023-06-23T00:40:08.718 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.712+0000 7f2cdd267700 1 -- 172.21.15.64:0/3454924769 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd000b870 con 0x7f2cd815d030 2023-06-23T00:40:08.718 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.712+0000 7f2cdd267700 1 -- 172.21.15.64:0/3454924769 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8001470 con 0x7f2cd815d030 2023-06-23T00:40:08.720 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.712+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3454924769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cd815d030 msgr2=0x7f2cd8068730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:08.720 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.712+0000 7f2cdf26b700 1 --2- 172.21.15.64:0/3454924769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cd815d030 0x7f2cd8068730 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0009f10 tx=0x7f2cd0009780 comp rx=0 tx=0).stop 2023-06-23T00:40:08.721 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.713+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3454924769 shutdown_connections 2023-06-23T00:40:08.721 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.713+0000 7f2cdf26b700 1 --2- 172.21.15.64:0/3454924769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cd815d030 0x7f2cd8068730 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:08.721 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.713+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3454924769 >> 172.21.15.64:0/3454924769 conn(0x7f2cd8069040 msgr2=0x7f2cd80b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:08.721 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.713+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3454924769 shutdown_connections 2023-06-23T00:40:08.721 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.713+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3454924769 wait complete. 2023-06-23T00:40:08.721 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.714+0000 7f2cdf26b700 1 Processor -- start 2023-06-23T00:40:08.721 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.714+0000 7f2cdf26b700 1 -- start start 2023-06-23T00:40:08.722 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.714+0000 7f2cdf26b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd80cf990 0x7f2cd8151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:08.722 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.714+0000 7f2cdf26b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd815d030 0x7f2cd80b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:08.722 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.714+0000 7f2cdf26b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cd815d400 0x7f2cd80b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:08.722 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cdf26b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd815a220 con 0x7f2cd80cf990 2023-06-23T00:40:08.722 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cdf26b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd814f510 con 0x7f2cd815d030 2023-06-23T00:40:08.722 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cdf26b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd8159c60 con 0x7f2cd815d400 2023-06-23T00:40:08.722 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cde269700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd80cf990 0x7f2cd8151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:08.722 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cde269700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd80cf990 0x7f2cd8151590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51564/0 (socket says 172.21.15.64:51564) 2023-06-23T00:40:08.723 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cde269700 1 -- 172.21.15.64:0/3567767564 learned_addr learned my addr 172.21.15.64:0/3567767564 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:08.723 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cdda68700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd815d030 0x7f2cd80b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:08.723 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cdea6a700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cd815d400 0x7f2cd80b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:08.723 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cde269700 1 -- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cd815d400 msgr2=0x7f2cd80b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:08.723 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cde269700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cd815d400 0x7f2cd80b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:08.723 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cde269700 1 -- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd815d030 msgr2=0x7f2cd80b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:08.723 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cde269700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd815d030 0x7f2cd80b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:08.724 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cde269700 1 -- 172.21.15.64:0/3567767564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd0009a20 con 0x7f2cd80cf990 2023-06-23T00:40:08.724 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cdda68700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd815d030 0x7f2cd80b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:08.724 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cde269700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd80cf990 0x7f2cd8151590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00097c0 tx=0x7f2cd0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:08.724 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cce7fc700 1 -- 172.21.15.64:0/3567767564 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0000f40 con 0x7f2cd80cf990 2023-06-23T00:40:08.724 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cce7fc700 1 -- 172.21.15.64:0/3567767564 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd0026400 con 0x7f2cd80cf990 2023-06-23T00:40:08.724 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3567767564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd8159f40 con 0x7f2cd80cf990 2023-06-23T00:40:08.724 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cce7fc700 1 -- 172.21.15.64:0/3567767564 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00267c0 con 0x7f2cd80cf990 2023-06-23T00:40:08.724 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.715+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3567767564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd8158fc0 con 0x7f2cd80cf990 2023-06-23T00:40:08.725 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.716+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3567767564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd814f000 con 0x7f2cd80cf990 2023-06-23T00:40:08.725 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.717+0000 7f2cce7fc700 1 -- 172.21.15.64:0/3567767564 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f2cd0019030 con 0x7f2cd80cf990 2023-06-23T00:40:08.725 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.717+0000 7f2cce7fc700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2cbc066e00 0x7f2cbc0692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:08.725 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.717+0000 7f2cce7fc700 1 -- 172.21.15.64:0/3567767564 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2227+0+0 (secure 0 0 0) 0x7f2cd0085030 con 0x7f2cd80cf990 2023-06-23T00:40:08.725 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.717+0000 7f2cdda68700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2cbc066e00 0x7f2cbc0692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:08.726 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.718+0000 7f2cdda68700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2cbc066e00 0x7f2cbc0692b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2cd80cd2e0 tx=0x7f2cc80019e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:08.726 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.721+0000 7f2cce7fc700 1 -- 172.21.15.64:0/3567767564 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f2cd814f000 con 0x7f2cd80cf990 2023-06-23T00:40:08.929 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.925+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3567767564 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cd814f000 con 0x7f2cd80cf990 2023-06-23T00:40:08.929 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.926+0000 7f2cce7fc700 1 -- 172.21.15.64:0/3567767564 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10396 (secure 0 0 0) 0x7f2cd814f000 con 0x7f2cd80cf990 2023-06-23T00:40:08.930 INFO:teuthology.misc.health.smithi064.stdout: 2023-06-23T00:40:08.930 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":13,"fsid":"72efde62-4c91-4277-87ee-356e50bd4cf5","created":"2023-06-23T00:39:58.078232+0000","modified":"2023-06-23T00:40:08.853479+0000","last_up_change":"2023-06-23T00:40:08.853479+0000","last_in_change":"2023-06-23T00:40:00.493656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":88027},{"type":"v1","addr":"172.21.15.64:6801","nonce":88027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":88027},{"type":"v1","addr":"172.21.15.64:6803","nonce":88027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":88027},{"type":"v1","addr":"172.21.15.64:6807","nonce":88027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":88027},{"type":"v1","addr":"172.21.15.64:6805","nonce":88027}]},"public_addr":"172.21.15.64:6801/88027","cluster_addr":"172.21.15.64:6803/88027","heartbeat_back_addr":"172.21.15.64:6807/88027","heartbeat_front_addr":"172.21.15.64:6805/88027","state":["exists","up"]},{"osd":1,"uuid":"f0698c9d-c67a-42ac-8d36-838836c514d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":88062},{"type":"v1","addr":"172.21.15.64:6825","nonce":88062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":88062},{"type":"v1","addr":"172.21.15.64:6827","nonce":88062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":88062},{"type":"v1","addr":"172.21.15.64:6831","nonce":88062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":88062},{"type":"v1","addr":"172.21.15.64:6829","nonce":88062}]},"public_addr":"172.21.15.64:6825/88062","cluster_addr":"172.21.15.64:6827/88062","heartbeat_back_addr":"172.21.15.64:6831/88062","heartbeat_front_addr":"172.21.15.64:6829/88062","state":["exists","up"]},{"osd":2,"uuid":"a5ece869-9da7-40e8-8f1d-bbaf67bb2187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":88045},{"type":"v1","addr":"172.21.15.64:6817","nonce":88045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":88045},{"type":"v1","addr":"172.21.15.64:6819","nonce":88045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":88045},{"type":"v1","addr":"172.21.15.64:6823","nonce":88045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":88045},{"type":"v1","addr":"172.21.15.64:6821","nonce":88045}]},"public_addr":"172.21.15.64:6817/88045","cluster_addr":"172.21.15.64:6819/88045","heartbeat_back_addr":"172.21.15.64:6823/88045","heartbeat_front_addr":"172.21.15.64:6821/88045","state":["exists","up"]},{"osd":3,"uuid":"21067fac-663b-4ecf-ae67-bf235161638e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":88044},{"type":"v1","addr":"172.21.15.64:6809","nonce":88044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":88044},{"type":"v1","addr":"172.21.15.64:6811","nonce":88044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":88044},{"type":"v1","addr":"172.21.15.64:6815","nonce":88044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":88044},{"type":"v1","addr":"172.21.15.64:6813","nonce":88044}]},"public_addr":"172.21.15.64:6809/88044","cluster_addr":"172.21.15.64:6811/88044","heartbeat_back_addr":"172.21.15.64:6815/88044","heartbeat_front_addr":"172.21.15.64:6813/88044","state":["exists","up"]},{"osd":4,"uuid":"6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":118144},{"type":"v1","addr":"172.21.15.86:6801","nonce":118144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":118144},{"type":"v1","addr":"172.21.15.86:6803","nonce":118144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":118144},{"type":"v1","addr":"172.21.15.86:6807","nonce":118144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":118144},{"type":"v1","addr":"172.21.15.86:6805","nonce":118144}]},"public_addr":"172.21.15.86:6801/118144","cluster_addr":"172.21.15.86:6803/118144","heartbeat_back_addr":"172.21.15.86:6807/118144","heartbeat_front_addr":"172.21.15.86:6805/118144","state":["exists","up"]},{"osd":5,"uuid":"0e987a73-8b36-4f26-94fd-f0135d6062c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":118169},{"type":"v1","addr":"172.21.15.86:6817","nonce":118169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":118169},{"type":"v1","addr":"172.21.15.86:6819","nonce":118169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":118169},{"type":"v1","addr":"172.21.15.86:6823","nonce":118169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":118169},{"type":"v1","addr":"172.21.15.86:6821","nonce":118169}]},"public_addr":"172.21.15.86:6817/118169","cluster_addr":"172.21.15.86:6819/118169","heartbeat_back_addr":"172.21.15.86:6823/118169","heartbeat_front_addr":"172.21.15.86:6821/118169","state":["exists","up"]},{"osd":6,"uuid":"7e40bfd5-0f9f-448c-8ccf-2f44bf82a622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":118175},{"type":"v1","addr":"172.21.15.86:6825","nonce":118175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":118175},{"type":"v1","addr":"172.21.15.86:6827","nonce":118175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":118175},{"type":"v1","addr":"172.21.15.86:6831","nonce":118175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":118175},{"type":"v1","addr":"172.21.15.86:6829","nonce":118175}]},"public_addr":"172.21.15.86:6825/118175","cluster_addr":"172.21.15.86:6827/118175","heartbeat_back_addr":"172.21.15.86:6831/118175","heartbeat_front_addr":"172.21.15.86:6829/118175","state":["exists","up"]},{"osd":7,"uuid":"2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":118145},{"type":"v1","addr":"172.21.15.86:6809","nonce":118145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":118145},{"type":"v1","addr":"172.21.15.86:6811","nonce":118145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":118145},{"type":"v1","addr":"172.21.15.86:6815","nonce":118145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":118145},{"type":"v1","addr":"172.21.15.86:6813","nonce":118145}]},"public_addr":"172.21.15.86:6809/118145","cluster_addr":"172.21.15.86:6811/118145","heartbeat_back_addr":"172.21.15.86:6815/118145","heartbeat_front_addr":"172.21.15.86:6813/118145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T00:40:08.932 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2cbc066e00 msgr2=0x7f2cbc0692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:08.932 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2cbc066e00 0x7f2cbc0692b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2cd80cd2e0 tx=0x7f2cc80019e0 comp rx=0 tx=0).stop 2023-06-23T00:40:08.932 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3567767564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd80cf990 msgr2=0x7f2cd8151590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:08.933 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd80cf990 0x7f2cd8151590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00097c0 tx=0x7f2cd0017000 comp rx=0 tx=0).stop 2023-06-23T00:40:08.933 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3567767564 shutdown_connections 2023-06-23T00:40:08.933 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2cbc066e00 0x7f2cbc0692b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:08.933 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cd815d400 0x7f2cd80b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:08.933 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd80cf990 0x7f2cd8151590 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:08.933 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 --2- 172.21.15.64:0/3567767564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cd815d030 0x7f2cd80b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:08.933 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3567767564 >> 172.21.15.64:0/3567767564 conn(0x7f2cd8069040 msgr2=0x7f2cd814df00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:08.933 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3567767564 shutdown_connections 2023-06-23T00:40:08.934 INFO:teuthology.misc.health.smithi064.stderr:2023-06-23T00:40:08.928+0000 7f2cdf26b700 1 -- 172.21.15.64:0/3567767564 wait complete. 2023-06-23T00:40:08.942 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-06-23T00:40:08.943 INFO:tasks.ceph:Creating RBD pool 2023-06-23T00:40:08.943 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool create rbd 8 2023-06-23T00:40:09.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.058+0000 7f0a7266c700 1 Processor -- start 2023-06-23T00:40:09.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.058+0000 7f0a7266c700 1 -- start start 2023-06-23T00:40:09.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a7266c700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f0a6c15d700 0x7f0a6c15dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:09.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a7266c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c15c4b0 0x7f0a6c15c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:09.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a7266c700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6c15e0e0 con 0x7f0a6c15d330 2023-06-23T00:40:09.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a7266c700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6c06a530 con 0x7f0a6c15d700 2023-06-23T00:40:09.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a7266c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c06adf0 con 0x7f0a6c15c4b0 2023-06-23T00:40:09.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a7166a700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f0a6c15d330 0x7f0a6c0b21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36610/0 (socket says 172.21.15.64:36610) 2023-06-23T00:40:09.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a7166a700 1 -- 172.21.15.64:0/2735571877 learned_addr learned my addr 172.21.15.64:0/2735571877 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:09.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a70e69700 1 --2- 172.21.15.64:0/2735571877 >> v2:172.21.15.64:3300/0 conn(0x7f0a6c15d700 0x7f0a6c15dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:09.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a71e6b700 1 --2- 172.21.15.64:0/2735571877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c15c4b0 0x7f0a6c15c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:09.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a6bfff700 1 -- 172.21.15.64:0/2735571877 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179011636 0 0) 0x7f0a6c15e0e0 con 0x7f0a6c15d330 2023-06-23T00:40:09.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a6bfff700 1 -- 172.21.15.64:0/2735571877 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a54003570 con 0x7f0a6c15d330 2023-06-23T00:40:09.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a70e69700 1 -- 172.21.15.64:0/2735571877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c15c4b0 msgr2=0x7f0a6c15c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:09.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a70e69700 1 --2- 172.21.15.64:0/2735571877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c15c4b0 0x7f0a6c15c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:09.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a70e69700 1 -- 172.21.15.64:0/2735571877 >> v1:172.21.15.64:6789/0 conn(0x7f0a6c15d330 legacy=0x7f0a6c0b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:09.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.059+0000 7f0a70e69700 1 -- 172.21.15.64:0/2735571877 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c06a7a0 con 0x7f0a6c15d700 2023-06-23T00:40:09.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a70e69700 1 --2- 172.21.15.64:0/2735571877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a6c15d700 0x7f0a6c15dba0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0a600098e0 tx=0x7f0a600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c563817c7a097e73 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:09.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a6bfff700 1 -- 172.21.15.64:0/2735571877 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60011630 con 0x7f0a6c15d700 2023-06-23T00:40:09.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a6bfff700 1 -- 172.21.15.64:0/2735571877 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a60011790 con 0x7f0a6c15d700 2023-06-23T00:40:09.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a6bfff700 1 -- 172.21.15.64:0/2735571877 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60011b30 con 0x7f0a6c15d700 2023-06-23T00:40:09.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a7266c700 1 -- 172.21.15.64:0/2735571877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a6c15d700 msgr2=0x7f0a6c15dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:09.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a7266c700 1 --2- 172.21.15.64:0/2735571877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a6c15d700 0x7f0a6c15dba0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0a600098e0 tx=0x7f0a600091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:09.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a7266c700 1 -- 172.21.15.64:0/2735571877 shutdown_connections 2023-06-23T00:40:09.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a7266c700 1 --2- 172.21.15.64:0/2735571877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c15c4b0 0x7f0a6c15c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:09.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a7266c700 1 --2- 172.21.15.64:0/2735571877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a6c15d700 0x7f0a6c15dba0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:09.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a7266c700 1 -- 172.21.15.64:0/2735571877 >> 172.21.15.64:0/2735571877 conn(0x7f0a6c0d0b30 msgr2=0x7f0a6c0ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:09.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a7266c700 1 -- 172.21.15.64:0/2735571877 shutdown_connections 2023-06-23T00:40:09.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.060+0000 7f0a7266c700 1 -- 172.21.15.64:0/2735571877 wait complete. 2023-06-23T00:40:09.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.061+0000 7f0a7266c700 1 Processor -- start 2023-06-23T00:40:09.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.061+0000 7f0a7266c700 1 -- start start 2023-06-23T00:40:09.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.062+0000 7f0a7266c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c15c4b0 0x7f0a6c15b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:09.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.062+0000 7f0a7266c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c15d330 0x7f0a6c16a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:09.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.062+0000 7f0a7266c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a6c15d700 0x7f0a6c0c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.062+0000 7f0a7266c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0c70d0 con 0x7f0a6c15d700 2023-06-23T00:40:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.062+0000 7f0a7266c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0c7f30 con 0x7f0a6c15d330 2023-06-23T00:40:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.062+0000 7f0a7266c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0c7320 con 0x7f0a6c15c4b0 2023-06-23T00:40:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.062+0000 7f0a71e6b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a6c15d700 0x7f0a6c0c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.062+0000 7f0a71e6b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a6c15d700 0x7f0a6c0c5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51580/0 (socket says 172.21.15.64:51580) 2023-06-23T00:40:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.062+0000 7f0a71e6b700 1 -- 172.21.15.64:0/4288623980 learned_addr learned my addr 172.21.15.64:0/4288623980 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.062+0000 7f0a7166a700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c15c4b0 0x7f0a6c15b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:09.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.062+0000 7f0a70e69700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c15d330 0x7f0a6c16a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:09.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.063+0000 7f0a71e6b700 1 -- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c15c4b0 msgr2=0x7f0a6c15b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:09.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.063+0000 7f0a71e6b700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c15c4b0 0x7f0a6c15b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:09.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.063+0000 7f0a71e6b700 1 -- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c15d330 msgr2=0x7f0a6c16a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:09.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.063+0000 7f0a71e6b700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c15d330 0x7f0a6c16a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:09.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.063+0000 7f0a71e6b700 1 -- 172.21.15.64:0/4288623980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60009420 con 0x7f0a6c15d700 2023-06-23T00:40:09.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.063+0000 7f0a71e6b700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a6c15d700 0x7f0a6c0c5b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009ec0 tx=0x7f0a5c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:09.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.063+0000 7f0a697fa700 1 -- 172.21.15.64:0/4288623980 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c00cb70 con 0x7f0a6c15d700 2023-06-23T00:40:09.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.063+0000 7f0a697fa700 1 -- 172.21.15.64:0/4288623980 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c00f040 con 0x7f0a6c15d700 2023-06-23T00:40:09.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.063+0000 7f0a7266c700 1 -- 172.21.15.64:0/4288623980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c0c7600 con 0x7f0a6c15d700 2023-06-23T00:40:09.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.063+0000 7f0a697fa700 1 -- 172.21.15.64:0/4288623980 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c014480 con 0x7f0a6c15d700 2023-06-23T00:40:09.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.063+0000 7f0a7266c700 1 -- 172.21.15.64:0/4288623980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c0c7bc0 con 0x7f0a6c15d700 2023-06-23T00:40:09.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.064+0000 7f0a7266c700 1 -- 172.21.15.64:0/4288623980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c04c1b0 con 0x7f0a6c15d700 2023-06-23T00:40:09.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.064+0000 7f0a697fa700 1 -- 172.21.15.64:0/4288623980 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f0a5c014830 con 0x7f0a6c15d700 2023-06-23T00:40:09.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.065+0000 7f0a697fa700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0a58066e00 0x7f0a580692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:09.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.065+0000 7f0a697fa700 1 -- 172.21.15.64:0/4288623980 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7f0a5c084760 con 0x7f0a6c15d700 2023-06-23T00:40:09.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.065+0000 7f0a7166a700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0a58066e00 0x7f0a580692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:09.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.065+0000 7f0a7166a700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0a58066e00 0x7f0a580692b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0a64007fd0 tx=0x7f0a64009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:09.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.069+0000 7f0a697fa700 1 -- 172.21.15.64:0/4288623980 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f0a5c052040 con 0x7f0a6c15d700 2023-06-23T00:40:09.265 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.262+0000 7f0a7266c700 1 -- 172.21.15.64:0/4288623980 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f0a6c0c8070 con 0x7f0a6c15d700 2023-06-23T00:40:09.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.963+0000 7f0a697fa700 1 -- 172.21.15.64:0/4288623980 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7f0a5c055660 con 0x7f0a6c15d700 2023-06-23T00:40:09.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 -- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0a58066e00 msgr2=0x7f0a580692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:09.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0a58066e00 0x7f0a580692b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0a64007fd0 tx=0x7f0a64009040 comp rx=0 tx=0).stop 2023-06-23T00:40:09.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 -- 172.21.15.64:0/4288623980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a6c15d700 msgr2=0x7f0a6c0c5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:09.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a6c15d700 0x7f0a6c0c5b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009ec0 tx=0x7f0a5c00b040 comp rx=0 tx=0).stop 2023-06-23T00:40:09.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 -- 172.21.15.64:0/4288623980 shutdown_connections 2023-06-23T00:40:09.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0a58066e00 0x7f0a580692b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:09.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0a6c15d700 0x7f0a6c0c5b80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:09.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a6c15c4b0 0x7f0a6c15b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:09.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 --2- 172.21.15.64:0/4288623980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c15d330 0x7f0a6c16a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:09.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 -- 172.21.15.64:0/4288623980 >> 172.21.15.64:0/4288623980 conn(0x7f0a6c0d0b30 msgr2=0x7f0a6c067550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:09.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 -- 172.21.15.64:0/4288623980 shutdown_connections 2023-06-23T00:40:09.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:09.966+0000 7f0a7266c700 1 -- 172.21.15.64:0/4288623980 wait complete. 2023-06-23T00:40:09.971 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'rbd' created 2023-06-23T00:40:09.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-06-23T00:40:10.108 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.105+0000 7f4bc4407700 1 Processor -- start 2023-06-23T00:40:10.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.105+0000 7f4bc4407700 1 -- start start 2023-06-23T00:40:10.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc4407700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f4bbc0ccd40 0x7f4bbc0cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:10.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc4407700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bbc0cf970 0x7f4bbc0cde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:10.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc4407700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc0cf4e0 con 0x7f4bbc0ce390 2023-06-23T00:40:10.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc4407700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bbc0ce8d0 con 0x7f4bbc0ccd40 2023-06-23T00:40:10.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc4407700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0cebb0 con 0x7f4bbc0cf970 2023-06-23T00:40:10.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc21a3700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f4bbc0ccd40 0x7f4bbc0cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:10.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc21a3700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f4bbc0ccd40 0x7f4bbc0cd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51588/0 (socket says 172.21.15.64:51588) 2023-06-23T00:40:10.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc21a3700 1 -- 172.21.15.64:0/281734077 learned_addr learned my addr 172.21.15.64:0/281734077 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:10.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc19a2700 1 --2- 172.21.15.64:0/281734077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bbc0cf970 0x7f4bbc0cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:10.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc21a3700 1 -- 172.21.15.64:0/281734077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bbc0cf970 msgr2=0x7f4bbc0cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:10.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc21a3700 1 --2- 172.21.15.64:0/281734077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bbc0cf970 0x7f4bbc0cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:10.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc21a3700 1 -- 172.21.15.64:0/281734077 >> v1:172.21.15.86:6789/0 conn(0x7f4bbc0ce390 legacy=0x7f4bbc0aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:10.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc21a3700 1 -- 172.21.15.64:0/281734077 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc0cee90 con 0x7f4bbc0ccd40 2023-06-23T00:40:10.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc19a2700 1 --2- 172.21.15.64:0/281734077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bbc0cf970 0x7f4bbc0cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T00:40:10.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc21a3700 1 --2- 172.21.15.64:0/281734077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bbc0ccd40 0x7f4bbc0cd110 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40099b0 tx=0x7f4bb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16830c31915ab20 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:10.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc11a1700 1 -- 172.21.15.64:0/281734077 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb400b820 con 0x7f4bbc0ccd40 2023-06-23T00:40:10.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc11a1700 1 -- 172.21.15.64:0/281734077 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb400b980 con 0x7f4bbc0ccd40 2023-06-23T00:40:10.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.106+0000 7f4bc11a1700 1 -- 172.21.15.64:0/281734077 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb400bd20 con 0x7f4bbc0ccd40 2023-06-23T00:40:10.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.107+0000 7f4bc4407700 1 -- 172.21.15.64:0/281734077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bbc0ccd40 msgr2=0x7f4bbc0cd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:10.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.107+0000 7f4bc4407700 1 --2- 172.21.15.64:0/281734077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bbc0ccd40 0x7f4bbc0cd110 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40099b0 tx=0x7f4bb4009250 comp rx=0 tx=0).stop 2023-06-23T00:40:10.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.107+0000 7f4bc4407700 1 -- 172.21.15.64:0/281734077 shutdown_connections 2023-06-23T00:40:10.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.107+0000 7f4bc4407700 1 --2- 172.21.15.64:0/281734077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bbc0ccd40 0x7f4bbc0cd110 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:10.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.107+0000 7f4bc4407700 1 --2- 172.21.15.64:0/281734077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bbc0cf970 0x7f4bbc0cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:10.114 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.107+0000 7f4bc4407700 1 -- 172.21.15.64:0/281734077 >> 172.21.15.64:0/281734077 conn(0x7f4bbc087c70 msgr2=0x7f4bbc068510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:10.114 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.107+0000 7f4bc4407700 1 -- 172.21.15.64:0/281734077 shutdown_connections 2023-06-23T00:40:10.114 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.107+0000 7f4bc4407700 1 -- 172.21.15.64:0/281734077 wait complete. 2023-06-23T00:40:10.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.108+0000 7f4bc4407700 1 Processor -- start 2023-06-23T00:40:10.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.108+0000 7f4bc4407700 1 -- start start 2023-06-23T00:40:10.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc4407700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bbc0ccd40 0x7f4bbc151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:10.116 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc4407700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bbc0ce390 0x7f4bbc15d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:10.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc4407700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bbc0cf970 0x7f4bbc0b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:10.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc4407700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0c7fe0 con 0x7f4bbc0cf970 2023-06-23T00:40:10.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc4407700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc152400 con 0x7f4bbc0ccd40 2023-06-23T00:40:10.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc4407700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0c73d0 con 0x7f4bbc0ce390 2023-06-23T00:40:10.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc29a4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bbc0cf970 0x7f4bbc0b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:10.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc29a4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bbc0cf970 0x7f4bbc0b41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51590/0 (socket says 172.21.15.64:51590) 2023-06-23T00:40:10.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc29a4700 1 -- 172.21.15.64:0/3106186478 learned_addr learned my addr 172.21.15.64:0/3106186478 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:10.117 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc19a2700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bbc0ce390 0x7f4bbc15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:10.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc21a3700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bbc0ccd40 0x7f4bbc151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:10.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc29a4700 1 -- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bbc0ce390 msgr2=0x7f4bbc15d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:10.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc29a4700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bbc0ce390 0x7f4bbc15d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:10.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc29a4700 1 -- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bbc0ccd40 msgr2=0x7f4bbc151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:10.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc29a4700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bbc0ccd40 0x7f4bbc151ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:10.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc29a4700 1 -- 172.21.15.64:0/3106186478 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb40094f0 con 0x7f4bbc0cf970 2023-06-23T00:40:10.118 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.109+0000 7f4bc29a4700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bbc0cf970 0x7f4bbc0b41a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00c930 tx=0x7f4bac0026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:10.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.110+0000 7f4bba7fc700 1 -- 172.21.15.64:0/3106186478 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac00ce70 con 0x7f4bbc0cf970 2023-06-23T00:40:10.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.110+0000 7f4bba7fc700 1 -- 172.21.15.64:0/3106186478 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac00a7e0 con 0x7f4bbc0cf970 2023-06-23T00:40:10.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.110+0000 7f4bc4407700 1 -- 172.21.15.64:0/3106186478 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bbc0c76b0 con 0x7f4bbc0cf970 2023-06-23T00:40:10.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.110+0000 7f4bba7fc700 1 -- 172.21.15.64:0/3106186478 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac00aba0 con 0x7f4bbc0cf970 2023-06-23T00:40:10.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.110+0000 7f4bc4407700 1 -- 172.21.15.64:0/3106186478 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bbc0c7c70 con 0x7f4bbc0cf970 2023-06-23T00:40:10.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.111+0000 7f4bc4407700 1 -- 172.21.15.64:0/3106186478 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bbc04c1b0 con 0x7f4bbc0cf970 2023-06-23T00:40:10.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.111+0000 7f4bba7fc700 1 -- 172.21.15.64:0/3106186478 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f4bac012420 con 0x7f4bbc0cf970 2023-06-23T00:40:10.119 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.112+0000 7f4bba7fc700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4ba8066e00 0x7f4ba80692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:10.120 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.112+0000 7f4bba7fc700 1 -- 172.21.15.64:0/3106186478 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 5184+0+0 (secure 0 0 0) 0x7f4bac084500 con 0x7f4bbc0cf970 2023-06-23T00:40:10.120 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.112+0000 7f4bc21a3700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4ba8066e00 0x7f4ba80692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:10.120 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.112+0000 7f4bc21a3700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4ba8066e00 0x7f4ba80692b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40099b0 tx=0x7f4bb4016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:10.120 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.117+0000 7f4bba7fc700 1 -- 172.21.15.64:0/3106186478 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f4bac051bd0 con 0x7f4bbc0cf970 2023-06-23T00:40:10.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.309+0000 7f4bc4407700 1 -- 172.21.15.64:0/3106186478 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4bbc0cf170 con 0x7f4bbc0cf970 2023-06-23T00:40:10.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.967+0000 7f4bba7fc700 1 -- 172.21.15.64:0/3106186478 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 173+0+0 (secure 0 0 0) 0x7f4bac0551f0 con 0x7f4bbc0cf970 2023-06-23T00:40:10.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.969+0000 7f4bc4407700 1 -- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4ba8066e00 msgr2=0x7f4ba80692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:10.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.969+0000 7f4bc4407700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4ba8066e00 0x7f4ba80692b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40099b0 tx=0x7f4bb4016000 comp rx=0 tx=0).stop 2023-06-23T00:40:10.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.969+0000 7f4bc4407700 1 -- 172.21.15.64:0/3106186478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bbc0cf970 msgr2=0x7f4bbc0b41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:10.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.969+0000 7f4bc4407700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bbc0cf970 0x7f4bbc0b41a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4bac00c930 tx=0x7f4bac0026e0 comp rx=0 tx=0).stop 2023-06-23T00:40:10.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.970+0000 7f4bc4407700 1 -- 172.21.15.64:0/3106186478 shutdown_connections 2023-06-23T00:40:10.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.970+0000 7f4bc4407700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4ba8066e00 0x7f4ba80692b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:10.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.970+0000 7f4bc4407700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4bbc0cf970 0x7f4bbc0b41a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:10.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.970+0000 7f4bc4407700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bbc0ccd40 0x7f4bbc151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:10.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.970+0000 7f4bc4407700 1 --2- 172.21.15.64:0/3106186478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bbc0ce390 0x7f4bbc15d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:10.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.970+0000 7f4bc4407700 1 -- 172.21.15.64:0/3106186478 >> 172.21.15.64:0/3106186478 conn(0x7f4bbc087c70 msgr2=0x7f4bbc0cb140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:10.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.970+0000 7f4bc4407700 1 -- 172.21.15.64:0/3106186478 shutdown_connections 2023-06-23T00:40:10.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:10.970+0000 7f4bc4407700 1 -- 172.21.15.64:0/3106186478 wait complete. 2023-06-23T00:40:10.975 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rbd' on pool 'rbd' 2023-06-23T00:40:10.986 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-06-23T00:40:10.987 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-23T00:40:10.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-23T00:40:10.990 INFO:tasks.ceph.mds.a:Started 2023-06-23T00:40:10.990 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-23T00:40:10.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-23T00:40:10.992 INFO:tasks.ceph.mds.c:Started 2023-06-23T00:40:10.992 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-23T00:40:10.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-23T00:40:10.995 INFO:tasks.ceph.mds.b:Started 2023-06-23T00:40:10.995 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-23T00:40:10.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-23T00:40:10.998 INFO:tasks.ceph.mds.d:Started 2023-06-23T00:40:10.998 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-06-23T00:40:10.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T00:40:11.073 INFO:tasks.ceph.mds.c.smithi064.stdout:starting mds.c at 2023-06-23T00:40:11.074 INFO:tasks.ceph.mds.a.smithi064.stdout:starting mds.a at 2023-06-23T00:40:11.085 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2023-06-23T00:40:11.086 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2023-06-23T00:40:11.137 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.135+0000 7f54c460f700 1 Processor -- start 2023-06-23T00:40:11.137 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.135+0000 7f54c460f700 1 -- start start 2023-06-23T00:40:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c460f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54bc0c82e0 0x7f54bc0c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c460f700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc15bb30 con 0x7f54bc0ac940 2023-06-23T00:40:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c460f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54bc069e40 con 0x7f54bc157000 2023-06-23T00:40:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c460f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc15af20 con 0x7f54bc0c82e0 2023-06-23T00:40:11.139 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c2bac700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f54bc0ac940 0x7f54bc06c000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36624/0 (socket says 172.21.15.64:36624) 2023-06-23T00:40:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c2bac700 1 -- 172.21.15.64:0/2041097115 learned_addr learned my addr 172.21.15.64:0/2041097115 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c1baa700 1 --2- 172.21.15.64:0/2041097115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54bc0c82e0 0x7f54bc0c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c13a9700 1 -- 172.21.15.64:0/2041097115 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438259929 0 0) 0x7f54bc069e40 con 0x7f54bc157000 2023-06-23T00:40:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c13a9700 1 -- 172.21.15.64:0/2041097115 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a8003570 con 0x7f54bc157000 2023-06-23T00:40:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c13a9700 1 -- 172.21.15.64:0/2041097115 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270130168 0 0) 0x7f54bc15bb30 con 0x7f54bc0ac940 2023-06-23T00:40:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c13a9700 1 -- 172.21.15.64:0/2041097115 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54bc069e40 con 0x7f54bc0ac940 2023-06-23T00:40:11.140 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c1baa700 1 -- 172.21.15.64:0/2041097115 >> v1:172.21.15.86:6789/0 conn(0x7f54bc157000 legacy=0x7f54bc069730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c1baa700 1 -- 172.21.15.64:0/2041097115 >> v1:172.21.15.64:6789/0 conn(0x7f54bc0ac940 legacy=0x7f54bc06c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c1baa700 1 -- 172.21.15.64:0/2041097115 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc15b200 con 0x7f54bc0c82e0 2023-06-23T00:40:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c1baa700 1 --2- 172.21.15.64:0/2041097115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54bc0c82e0 0x7f54bc0c6d20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009910 tx=0x7f54ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dcadb0b6abca139 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c13a9700 1 -- 172.21.15.64:0/2041097115 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac011630 con 0x7f54bc0c82e0 2023-06-23T00:40:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c13a9700 1 -- 172.21.15.64:0/2041097115 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ac011790 con 0x7f54bc0c82e0 2023-06-23T00:40:11.141 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.136+0000 7f54c13a9700 1 -- 172.21.15.64:0/2041097115 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac011b10 con 0x7f54bc0c82e0 2023-06-23T00:40:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.137+0000 7f54c460f700 1 -- 172.21.15.64:0/2041097115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54bc0c82e0 msgr2=0x7f54bc0c6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.137+0000 7f54c460f700 1 --2- 172.21.15.64:0/2041097115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54bc0c82e0 0x7f54bc0c6d20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009910 tx=0x7f54ac0091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.137+0000 7f54c460f700 1 -- 172.21.15.64:0/2041097115 shutdown_connections 2023-06-23T00:40:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.137+0000 7f54c460f700 1 --2- 172.21.15.64:0/2041097115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54bc0c82e0 0x7f54bc0c6d20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.137+0000 7f54c460f700 1 -- 172.21.15.64:0/2041097115 >> 172.21.15.64:0/2041097115 conn(0x7f54bc0c9150 msgr2=0x7f54bc0c9550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.137+0000 7f54c460f700 1 -- 172.21.15.64:0/2041097115 shutdown_connections 2023-06-23T00:40:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.137+0000 7f54c460f700 1 -- 172.21.15.64:0/2041097115 wait complete. 2023-06-23T00:40:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.138+0000 7f54c460f700 1 Processor -- start 2023-06-23T00:40:11.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.138+0000 7f54c460f700 1 -- start start 2023-06-23T00:40:11.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c460f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54bc0ac940 0x7f54bc0c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c460f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54bc0c82e0 0x7f54bc159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c460f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54bc157000 0x7f54bc0b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c460f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc15a3c0 con 0x7f54bc157000 2023-06-23T00:40:11.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c460f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc0b0d80 con 0x7f54bc0c82e0 2023-06-23T00:40:11.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c460f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc0b0f70 con 0x7f54bc0ac940 2023-06-23T00:40:11.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c2bac700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54bc157000 0x7f54bc0b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c2bac700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54bc157000 0x7f54bc0b0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51636/0 (socket says 172.21.15.64:51636) 2023-06-23T00:40:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c2bac700 1 -- 172.21.15.64:0/1114818900 learned_addr learned my addr 172.21.15.64:0/1114818900 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c1baa700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54bc0c82e0 0x7f54bc159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c23ab700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54bc0ac940 0x7f54bc0c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c2bac700 1 -- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54bc0ac940 msgr2=0x7f54bc0c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c2bac700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54bc0ac940 0x7f54bc0c1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c2bac700 1 -- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54bc0c82e0 msgr2=0x7f54bc159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c2bac700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54bc0c82e0 0x7f54bc159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c2bac700 1 -- 172.21.15.64:0/1114818900 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac009420 con 0x7f54bc157000 2023-06-23T00:40:11.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.139+0000 7f54c2bac700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54bc157000 0x7f54bc0b0840 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f54b4001460 tx=0x7f54b4001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:11.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.140+0000 7f54b27fc700 1 -- 172.21.15.64:0/1114818900 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b400a890 con 0x7f54bc157000 2023-06-23T00:40:11.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.140+0000 7f54b27fc700 1 -- 172.21.15.64:0/1114818900 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b400a9f0 con 0x7f54bc157000 2023-06-23T00:40:11.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.140+0000 7f54c460f700 1 -- 172.21.15.64:0/1114818900 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54bc0b1250 con 0x7f54bc157000 2023-06-23T00:40:11.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.140+0000 7f54b27fc700 1 -- 172.21.15.64:0/1114818900 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b400ad70 con 0x7f54bc157000 2023-06-23T00:40:11.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.140+0000 7f54c460f700 1 -- 172.21.15.64:0/1114818900 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc0b1810 con 0x7f54bc157000 2023-06-23T00:40:11.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.140+0000 7f54c460f700 1 -- 172.21.15.64:0/1114818900 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54bc04c1b0 con 0x7f54bc157000 2023-06-23T00:40:11.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.141+0000 7f54b27fc700 1 -- 172.21.15.64:0/1114818900 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f54b4007300 con 0x7f54bc157000 2023-06-23T00:40:11.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.141+0000 7f54b27fc700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f54a0066e00 0x7f54a00692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.142+0000 7f54b27fc700 1 -- 172.21.15.64:0/1114818900 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f54b4084590 con 0x7f54bc157000 2023-06-23T00:40:11.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.142+0000 7f54c23ab700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f54a0066e00 0x7f54a00692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.142+0000 7f54c23ab700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f54a0066e00 0x7f54a00692b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f54b8000c00 tx=0x7f54b8007d00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:11.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.146+0000 7f54b27fc700 1 -- 172.21.15.64:0/1114818900 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f54b4051ae0 con 0x7f54bc157000 2023-06-23T00:40:11.359 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.356+0000 7f54c460f700 1 -- 172.21.15.64:0/1114818900 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54bc15b4e0 con 0x7f54bc157000 2023-06-23T00:40:11.359 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.357+0000 7f54b27fc700 1 -- 172.21.15.64:0/1114818900 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+13674 (secure 0 0 0) 0x7f54b4055100 con 0x7f54bc157000 2023-06-23T00:40:11.360 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:11.360 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":15,"fsid":"72efde62-4c91-4277-87ee-356e50bd4cf5","created":"2023-06-23T00:39:58.078232+0000","modified":"2023-06-23T00:40:10.960706+0000","last_up_change":"2023-06-23T00:40:08.853479+0000","last_in_change":"2023-06-23T00:40:00.493656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-23T00:40:09.294447+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-23T00:40:09.719568+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":88027},{"type":"v1","addr":"172.21.15.64:6801","nonce":88027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":88027},{"type":"v1","addr":"172.21.15.64:6803","nonce":88027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":88027},{"type":"v1","addr":"172.21.15.64:6807","nonce":88027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":88027},{"type":"v1","addr":"172.21.15.64:6805","nonce":88027}]},"public_addr":"172.21.15.64:6801/88027","cluster_addr":"172.21.15.64:6803/88027","heartbeat_back_addr":"172.21.15.64:6807/88027","heartbeat_front_addr":"172.21.15.64:6805/88027","state":["exists","up"]},{"osd":1,"uuid":"f0698c9d-c67a-42ac-8d36-838836c514d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":88062},{"type":"v1","addr":"172.21.15.64:6825","nonce":88062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":88062},{"type":"v1","addr":"172.21.15.64:6827","nonce":88062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":88062},{"type":"v1","addr":"172.21.15.64:6831","nonce":88062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":88062},{"type":"v1","addr":"172.21.15.64:6829","nonce":88062}]},"public_addr":"172.21.15.64:6825/88062","cluster_addr":"172.21.15.64:6827/88062","heartbeat_back_addr":"172.21.15.64:6831/88062","heartbeat_front_addr":"172.21.15.64:6829/88062","state":["exists","up"]},{"osd":2,"uuid":"a5ece869-9da7-40e8-8f1d-bbaf67bb2187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":88045},{"type":"v1","addr":"172.21.15.64:6817","nonce":88045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":88045},{"type":"v1","addr":"172.21.15.64:6819","nonce":88045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":88045},{"type":"v1","addr":"172.21.15.64:6823","nonce":88045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":88045},{"type":"v1","addr":"172.21.15.64:6821","nonce":88045}]},"public_addr":"172.21.15.64:6817/88045","cluster_addr":"172.21.15.64:6819/88045","heartbeat_back_addr":"172.21.15.64:6823/88045","heartbeat_front_addr":"172.21.15.64:6821/88045","state":["exists","up"]},{"osd":3,"uuid":"21067fac-663b-4ecf-ae67-bf235161638e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":88044},{"type":"v1","addr":"172.21.15.64:6809","nonce":88044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":88044},{"type":"v1","addr":"172.21.15.64:6811","nonce":88044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":88044},{"type":"v1","addr":"172.21.15.64:6815","nonce":88044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":88044},{"type":"v1","addr":"172.21.15.64:6813","nonce":88044}]},"public_addr":"172.21.15.64:6809/88044","cluster_addr":"172.21.15.64:6811/88044","heartbeat_back_addr":"172.21.15.64:6815/88044","heartbeat_front_addr":"172.21.15.64:6813/88044","state":["exists","up"]},{"osd":4,"uuid":"6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":118144},{"type":"v1","addr":"172.21.15.86:6801","nonce":118144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":118144},{"type":"v1","addr":"172.21.15.86:6803","nonce":118144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":118144},{"type":"v1","addr":"172.21.15.86:6807","nonce":118144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":118144},{"type":"v1","addr":"172.21.15.86:6805","nonce":118144}]},"public_addr":"172.21.15.86:6801/118144","cluster_addr":"172.21.15.86:6803/118144","heartbeat_back_addr":"172.21.15.86:6807/118144","heartbeat_front_addr":"172.21.15.86:6805/118144","state":["exists","up"]},{"osd":5,"uuid":"0e987a73-8b36-4f26-94fd-f0135d6062c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":118169},{"type":"v1","addr":"172.21.15.86:6817","nonce":118169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":118169},{"type":"v1","addr":"172.21.15.86:6819","nonce":118169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":118169},{"type":"v1","addr":"172.21.15.86:6823","nonce":118169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":118169},{"type":"v1","addr":"172.21.15.86:6821","nonce":118169}]},"public_addr":"172.21.15.86:6817/118169","cluster_addr":"172.21.15.86:6819/118169","heartbeat_back_addr":"172.21.15.86:6823/118169","heartbeat_front_addr":"172.21.15.86:6821/118169","state":["exists","up"]},{"osd":6,"uuid":"7e40bfd5-0f9f-448c-8ccf-2f44bf82a622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":118175},{"type":"v1","addr":"172.21.15.86:6825","nonce":118175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":118175},{"type":"v1","addr":"172.21.15.86:6827","nonce":118175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":118175},{"type":"v1","addr":"172.21.15.86:6831","nonce":118175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":118175},{"type":"v1","addr":"172.21.15.86:6829","nonce":118175}]},"public_addr":"172.21.15.86:6825/118175","cluster_addr":"172.21.15.86:6827/118175","heartbeat_back_addr":"172.21.15.86:6831/118175","heartbeat_front_addr":"172.21.15.86:6829/118175","state":["exists","up"]},{"osd":7,"uuid":"2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":118145},{"type":"v1","addr":"172.21.15.86:6809","nonce":118145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":118145},{"type":"v1","addr":"172.21.15.86:6811","nonce":118145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":118145},{"type":"v1","addr":"172.21.15.86:6815","nonce":118145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":118145},{"type":"v1","addr":"172.21.15.86:6813","nonce":118145}]},"public_addr":"172.21.15.86:6809/118145","cluster_addr":"172.21.15.86:6811/118145","heartbeat_back_addr":"172.21.15.86:6815/118145","heartbeat_front_addr":"172.21.15.86:6813/118145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.319777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.692469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.685180+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.648752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.461468+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.727692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.701726+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.703200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T00:40:11.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.359+0000 7f54c460f700 1 -- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f54a0066e00 msgr2=0x7f54a00692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.362 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.359+0000 7f54c460f700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f54a0066e00 0x7f54a00692b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f54b8000c00 tx=0x7f54b8007d00 comp rx=0 tx=0).stop 2023-06-23T00:40:11.362 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.359+0000 7f54c460f700 1 -- 172.21.15.64:0/1114818900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54bc157000 msgr2=0x7f54bc0b0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.362 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.359+0000 7f54c460f700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54bc157000 0x7f54bc0b0840 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f54b4001460 tx=0x7f54b4001ae0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.362 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.359+0000 7f54c460f700 1 -- 172.21.15.64:0/1114818900 shutdown_connections 2023-06-23T00:40:11.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.359+0000 7f54c460f700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f54a0066e00 0x7f54a00692b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.359+0000 7f54c460f700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54bc157000 0x7f54bc0b0840 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.359+0000 7f54c460f700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54bc0ac940 0x7f54bc0c1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.359+0000 7f54c460f700 1 --2- 172.21.15.64:0/1114818900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54bc0c82e0 0x7f54bc159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.359+0000 7f54c460f700 1 -- 172.21.15.64:0/1114818900 >> 172.21.15.64:0/1114818900 conn(0x7f54bc0c9150 msgr2=0x7f54bc154ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:11.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.360+0000 7f54c460f700 1 -- 172.21.15.64:0/1114818900 shutdown_connections 2023-06-23T00:40:11.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.360+0000 7f54c460f700 1 -- 172.21.15.64:0/1114818900 wait complete. 2023-06-23T00:40:11.374 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-06-23T00:40:09.294447+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-06-23T00:40:09.719568+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-23T00:40:11.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-23T00:40:11.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.497+0000 7f9f3f2ed700 1 Processor -- start 2023-06-23T00:40:11.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.497+0000 7f9f3f2ed700 1 -- start start 2023-06-23T00:40:11.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3f2ed700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9f380aef00 0x7f9f38155b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3f2ed700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f9f38156060 0x7f9f380b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3f2ed700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f38157830 con 0x7f9f380aeb30 2023-06-23T00:40:11.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3f2ed700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f380b2980 con 0x7f9f38156060 2023-06-23T00:40:11.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3f2ed700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f38159c60 con 0x7f9f380aef00 2023-06-23T00:40:11.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3d88a700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f9f38156060 0x7f9f380b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3d88a700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f9f38156060 0x7f9f380b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51650/0 (socket says 172.21.15.64:51650) 2023-06-23T00:40:11.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3d88a700 1 -- 172.21.15.64:0/2953439785 learned_addr learned my addr 172.21.15.64:0/2953439785 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:11.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3c888700 1 --2- 172.21.15.64:0/2953439785 >> v2:172.21.15.86:3300/0 conn(0x7f9f380aef00 0x7f9f38155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3d88a700 1 -- 172.21.15.64:0/2953439785 >> v2:172.21.15.86:3300/0 conn(0x7f9f380aef00 msgr2=0x7f9f38155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:11.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3d88a700 1 --2- 172.21.15.64:0/2953439785 >> v2:172.21.15.86:3300/0 conn(0x7f9f380aef00 0x7f9f38155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3d88a700 1 -- 172.21.15.64:0/2953439785 >> v1:172.21.15.86:6789/0 conn(0x7f9f380aeb30 legacy=0x7f9f380b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:11.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3d88a700 1 -- 172.21.15.64:0/2953439785 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f38159f40 con 0x7f9f38156060 2023-06-23T00:40:11.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f3d88a700 1 --2- 172.21.15.64:0/2953439785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f38156060 0x7f9f380b0310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f380cdfb0 tx=0x7f9f30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a0bec510195a6f server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:11.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.498+0000 7f9f2ffff700 1 -- 172.21.15.64:0/2953439785 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3000e070 con 0x7f9f38156060 2023-06-23T00:40:11.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.499+0000 7f9f2ffff700 1 -- 172.21.15.64:0/2953439785 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3000ba10 con 0x7f9f38156060 2023-06-23T00:40:11.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.499+0000 7f9f2ffff700 1 -- 172.21.15.64:0/2953439785 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f300092e0 con 0x7f9f38156060 2023-06-23T00:40:11.505 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.499+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/2953439785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f38156060 msgr2=0x7f9f380b0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.499+0000 7f9f3f2ed700 1 --2- 172.21.15.64:0/2953439785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f38156060 0x7f9f380b0310 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9f380cdfb0 tx=0x7f9f30009840 comp rx=0 tx=0).stop 2023-06-23T00:40:11.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.499+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/2953439785 shutdown_connections 2023-06-23T00:40:11.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.499+0000 7f9f3f2ed700 1 --2- 172.21.15.64:0/2953439785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f38156060 0x7f9f380b0310 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.499+0000 7f9f3f2ed700 1 --2- 172.21.15.64:0/2953439785 >> v2:172.21.15.86:3300/0 conn(0x7f9f380aef00 0x7f9f38155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.499+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/2953439785 >> 172.21.15.64:0/2953439785 conn(0x7f9f3806a0c0 msgr2=0x7f9f3806b100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:11.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.499+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/2953439785 shutdown_connections 2023-06-23T00:40:11.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.499+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/2953439785 wait complete. 2023-06-23T00:40:11.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.500+0000 7f9f3f2ed700 1 Processor -- start 2023-06-23T00:40:11.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.500+0000 7f9f3f2ed700 1 -- start start 2023-06-23T00:40:11.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.501+0000 7f9f3f2ed700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f380aeb30 0x7f9f38153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.501+0000 7f9f3f2ed700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f380aef00 0x7f9f38154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.501+0000 7f9f3f2ed700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f38156060 0x7f9f381550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.501+0000 7f9f3f2ed700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3816a530 con 0x7f9f380aef00 2023-06-23T00:40:11.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.501+0000 7f9f3f2ed700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f38169660 con 0x7f9f38156060 2023-06-23T00:40:11.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.501+0000 7f9f3f2ed700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f38169920 con 0x7f9f380aeb30 2023-06-23T00:40:11.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.501+0000 7f9f3c888700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f380aef00 0x7f9f38154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.501+0000 7f9f3c888700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f380aef00 0x7f9f38154bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51662/0 (socket says 172.21.15.64:51662) 2023-06-23T00:40:11.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.501+0000 7f9f3c888700 1 -- 172.21.15.64:0/1610784862 learned_addr learned my addr 172.21.15.64:0/1610784862 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:11.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f3d88a700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f38156060 0x7f9f381550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f3d089700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f380aeb30 0x7f9f38153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f3c888700 1 -- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f380aeb30 msgr2=0x7f9f38153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f3c888700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f380aeb30 0x7f9f38153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f3c888700 1 -- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f38156060 msgr2=0x7f9f381550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f3c888700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f38156060 0x7f9f381550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f3c888700 1 -- 172.21.15.64:0/1610784862 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f30009ae0 con 0x7f9f380aef00 2023-06-23T00:40:11.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f3d88a700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f38156060 0x7f9f381550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T00:40:11.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f3c888700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f380aef00 0x7f9f38154bb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9f2800d890 tx=0x7f9f28008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:11.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f2d7fa700 1 -- 172.21.15.64:0/1610784862 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2800b600 con 0x7f9f380aef00 2023-06-23T00:40:11.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f2d7fa700 1 -- 172.21.15.64:0/1610784862 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2800b760 con 0x7f9f380aef00 2023-06-23T00:40:11.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/1610784862 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f38169c00 con 0x7f9f380aef00 2023-06-23T00:40:11.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f2d7fa700 1 -- 172.21.15.64:0/1610784862 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2800bae0 con 0x7f9f380aef00 2023-06-23T00:40:11.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.502+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/1610784862 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3816a1c0 con 0x7f9f380aef00 2023-06-23T00:40:11.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.503+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/1610784862 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3816a6a0 con 0x7f9f380aef00 2023-06-23T00:40:11.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.504+0000 7f9f2d7fa700 1 -- 172.21.15.64:0/1610784862 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f9f2801e460 con 0x7f9f380aef00 2023-06-23T00:40:11.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.504+0000 7f9f2d7fa700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9f1c066e00 0x7f9f1c0692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.504+0000 7f9f2d7fa700 1 -- 172.21.15.64:0/1610784862 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f9f280847a0 con 0x7f9f380aef00 2023-06-23T00:40:11.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.505+0000 7f9f3d089700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9f1c066e00 0x7f9f1c0692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.505+0000 7f9f3d089700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9f1c066e00 0x7f9f1c0692b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f34006fd0 tx=0x7f9f34008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:11.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.509+0000 7f9f2d7fa700 1 -- 172.21.15.64:0/1610784862 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f9f28051da0 con 0x7f9f380aef00 2023-06-23T00:40:11.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.711+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/1610784862 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9f3804c1b0 con 0x7f9f380aef00 2023-06-23T00:40:11.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.711+0000 7f9f2d7fa700 1 -- 172.21.15.64:0/1610784862 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9f280553c0 con 0x7f9f380aef00 2023-06-23T00:40:11.714 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 8 2023-06-23T00:40:11.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9f1c066e00 msgr2=0x7f9f1c0692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9f1c066e00 0x7f9f1c0692b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f34006fd0 tx=0x7f9f34008040 comp rx=0 tx=0).stop 2023-06-23T00:40:11.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/1610784862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f380aef00 msgr2=0x7f9f38154bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f380aef00 0x7f9f38154bb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9f2800d890 tx=0x7f9f28008040 comp rx=0 tx=0).stop 2023-06-23T00:40:11.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/1610784862 shutdown_connections 2023-06-23T00:40:11.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9f1c066e00 0x7f9f1c0692b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f38156060 0x7f9f381550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f380aeb30 0x7f9f38153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 --2- 172.21.15.64:0/1610784862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f380aef00 0x7f9f38154bb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/1610784862 >> 172.21.15.64:0/1610784862 conn(0x7f9f3806a0c0 msgr2=0x7f9f380affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:11.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/1610784862 shutdown_connections 2023-06-23T00:40:11.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.713+0000 7f9f3f2ed700 1 -- 172.21.15.64:0/1610784862 wait complete. 2023-06-23T00:40:11.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T00:40:11.850 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.847+0000 7f3f01c55700 1 Processor -- start 2023-06-23T00:40:11.850 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.847+0000 7f3f01c55700 1 -- start start 2023-06-23T00:40:11.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3f01c55700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3efc157130 0x7f3efc0c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3f01c55700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3efc0b1660 0x7f3efc06df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3f01c55700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3efc15b520 con 0x7f3efc0ae150 2023-06-23T00:40:11.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3f01c55700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3efc0c93c0 con 0x7f3efc0b1660 2023-06-23T00:40:11.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3f01c55700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3efc15aed0 con 0x7f3efc157130 2023-06-23T00:40:11.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3efbfff700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3efc0b1660 0x7f3efc06df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3efbfff700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3efc0b1660 0x7f3efc06df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51666/0 (socket says 172.21.15.64:51666) 2023-06-23T00:40:11.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3efbfff700 1 -- 172.21.15.64:0/1786585802 learned_addr learned my addr 172.21.15.64:0/1786585802 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:11.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3efb7fe700 1 --2- 172.21.15.64:0/1786585802 >> v2:172.21.15.86:3300/0 conn(0x7f3efc157130 0x7f3efc0c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3efbfff700 1 -- 172.21.15.64:0/1786585802 >> v2:172.21.15.86:3300/0 conn(0x7f3efc157130 msgr2=0x7f3efc0c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:11.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3efbfff700 1 --2- 172.21.15.64:0/1786585802 >> v2:172.21.15.86:3300/0 conn(0x7f3efc157130 0x7f3efc0c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3efbfff700 1 -- 172.21.15.64:0/1786585802 >> v1:172.21.15.86:6789/0 conn(0x7f3efc0ae150 legacy=0x7f3efc0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.848+0000 7f3efbfff700 1 -- 172.21.15.64:0/1786585802 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc15b1b0 con 0x7f3efc0b1660 2023-06-23T00:40:11.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3efbfff700 1 --2- 172.21.15.64:0/1786585802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0b1660 0x7f3efc06df70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3efc0643a0 tx=0x7f3ef4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3f83d1d6d25658 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:11.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3efa7fc700 1 -- 172.21.15.64:0/1786585802 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef400e070 con 0x7f3efc0b1660 2023-06-23T00:40:11.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3efa7fc700 1 -- 172.21.15.64:0/1786585802 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef400ba20 con 0x7f3efc0b1660 2023-06-23T00:40:11.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3efa7fc700 1 -- 172.21.15.64:0/1786585802 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef40092e0 con 0x7f3efc0b1660 2023-06-23T00:40:11.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3f01c55700 1 -- 172.21.15.64:0/1786585802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0b1660 msgr2=0x7f3efc06df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3f01c55700 1 --2- 172.21.15.64:0/1786585802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0b1660 0x7f3efc06df70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3efc0643a0 tx=0x7f3ef4009840 comp rx=0 tx=0).stop 2023-06-23T00:40:11.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3f01c55700 1 -- 172.21.15.64:0/1786585802 shutdown_connections 2023-06-23T00:40:11.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3f01c55700 1 --2- 172.21.15.64:0/1786585802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0b1660 0x7f3efc06df70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3f01c55700 1 --2- 172.21.15.64:0/1786585802 >> v2:172.21.15.86:3300/0 conn(0x7f3efc157130 0x7f3efc0c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3f01c55700 1 -- 172.21.15.64:0/1786585802 >> 172.21.15.64:0/1786585802 conn(0x7f3efc068fb0 msgr2=0x7f3efc0ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:11.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3f01c55700 1 -- 172.21.15.64:0/1786585802 shutdown_connections 2023-06-23T00:40:11.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.849+0000 7f3f01c55700 1 -- 172.21.15.64:0/1786585802 wait complete. 2023-06-23T00:40:11.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.851+0000 7f3f01c55700 1 Processor -- start 2023-06-23T00:40:11.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.851+0000 7f3f01c55700 1 -- start start 2023-06-23T00:40:11.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.851+0000 7f3f01c55700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3efc0ae150 0x7f3efc0a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.851+0000 7f3f01c55700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0b1660 0x7f3efc0b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.851+0000 7f3f01c55700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3efc157130 0x7f3efc157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.851+0000 7f3f01c55700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc0aa260 con 0x7f3efc0b1660 2023-06-23T00:40:11.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.851+0000 7f3f01c55700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc14b300 con 0x7f3efc0ae150 2023-06-23T00:40:11.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.851+0000 7f3f01c55700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3efc0a8c80 con 0x7f3efc157130 2023-06-23T00:40:11.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efaffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0b1660 0x7f3efc0b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3efc0ae150 0x7f3efc0a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efb7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3efc0ae150 0x7f3efc0a9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.64:59282/0 (socket says 172.21.15.64:59282) 2023-06-23T00:40:11.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efb7fe700 1 -- 172.21.15.64:0/33607926 learned_addr learned my addr 172.21.15.64:0/33607926 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:11.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efbfff700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3efc157130 0x7f3efc157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efaffd700 1 -- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3efc157130 msgr2=0x7f3efc157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efaffd700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3efc157130 0x7f3efc157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efaffd700 1 -- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3efc0ae150 msgr2=0x7f3efc0a9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:11.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efaffd700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3efc0ae150 0x7f3efc0a9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:11.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efaffd700 1 -- 172.21.15.64:0/33607926 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef4009ae0 con 0x7f3efc0b1660 2023-06-23T00:40:11.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efb7fe700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3efc0ae150 0x7f3efc0a9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:11.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3efaffd700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0b1660 0x7f3efc0b05c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000c590 tx=0x7f3ef0007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:11.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3ee37fe700 1 -- 172.21.15.64:0/33607926 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0011590 con 0x7f3efc0b1660 2023-06-23T00:40:11.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3ee37fe700 1 -- 172.21.15.64:0/33607926 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef0011710 con 0x7f3efc0b1660 2023-06-23T00:40:11.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3f01c55700 1 -- 172.21.15.64:0/33607926 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3efc0a8f60 con 0x7f3efc0b1660 2023-06-23T00:40:11.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3ee37fe700 1 -- 172.21.15.64:0/33607926 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0011ad0 con 0x7f3efc0b1660 2023-06-23T00:40:11.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.852+0000 7f3f01c55700 1 -- 172.21.15.64:0/33607926 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3efc0a9520 con 0x7f3efc0b1660 2023-06-23T00:40:11.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.853+0000 7f3f01c55700 1 -- 172.21.15.64:0/33607926 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3efc04c1b0 con 0x7f3efc0b1660 2023-06-23T00:40:11.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.854+0000 7f3ee37fe700 1 -- 172.21.15.64:0/33607926 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f3ef001b3b0 con 0x7f3efc0b1660 2023-06-23T00:40:11.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.854+0000 7f3ee37fe700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ee8066e00 0x7f3ee80692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:11.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.854+0000 7f3ee37fe700 1 -- 172.21.15.64:0/33607926 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3ef0083400 con 0x7f3efc0b1660 2023-06-23T00:40:11.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.854+0000 7f3efb7fe700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ee8066e00 0x7f3ee80692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:11.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.855+0000 7f3efb7fe700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ee8066e00 0x7f3ee80692b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009fd0 tx=0x7f3eec0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:11.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:11.858+0000 7f3ee37fe700 1 -- 172.21.15.64:0/33607926 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f3ef0050a00 con 0x7f3efc0b1660 2023-06-23T00:40:12.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.053+0000 7f3f01c55700 1 -- 172.21.15.64:0/33607926 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3efc0c96a0 con 0x7f3efc0b1660 2023-06-23T00:40:12.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.053+0000 7f3ee37fe700 1 -- 172.21.15.64:0/33607926 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3ef0054020 con 0x7f3efc0b1660 2023-06-23T00:40:12.056 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-06-23T00:40:12.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 -- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ee8066e00 msgr2=0x7f3ee80692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ee8066e00 0x7f3ee80692b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009fd0 tx=0x7f3eec0091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 -- 172.21.15.64:0/33607926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0b1660 msgr2=0x7f3efc0b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0b1660 0x7f3efc0b05c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000c590 tx=0x7f3ef0007ec0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 -- 172.21.15.64:0/33607926 shutdown_connections 2023-06-23T00:40:12.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ee8066e00 0x7f3ee80692b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3efc157130 0x7f3efc157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.059 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3efc0ae150 0x7f3efc0a9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.059 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 --2- 172.21.15.64:0/33607926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3efc0b1660 0x7f3efc0b05c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.059 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 -- 172.21.15.64:0/33607926 >> 172.21.15.64:0/33607926 conn(0x7f3efc068fb0 msgr2=0x7f3efc0b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:12.059 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 -- 172.21.15.64:0/33607926 shutdown_connections 2023-06-23T00:40:12.059 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.055+0000 7f3f01c55700 1 -- 172.21.15.64:0/33607926 wait complete. 2023-06-23T00:40:12.070 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T00:40:12.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.189+0000 7fe655dfd700 1 Processor -- start 2023-06-23T00:40:12.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.189+0000 7fe655dfd700 1 -- start start 2023-06-23T00:40:12.194 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.190+0000 7fe655dfd700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fe650068b30 0x7fe6500c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.194 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.190+0000 7fe655dfd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe65015c3e0 0x7fe65015c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.194 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.190+0000 7fe655dfd700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6500c8350 con 0x7fe650068760 2023-06-23T00:40:12.194 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.190+0000 7fe655dfd700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6500c7810 con 0x7fe650068b30 2023-06-23T00:40:12.194 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.190+0000 7fe655dfd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65006d020 con 0x7fe65015c3e0 2023-06-23T00:40:12.195 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.190+0000 7fe64effd700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fe650068b30 0x7fe6500c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.195 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.190+0000 7fe64effd700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fe650068b30 0x7fe6500c87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51688/0 (socket says 172.21.15.64:51688) 2023-06-23T00:40:12.195 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.190+0000 7fe64effd700 1 -- 172.21.15.64:0/4195735913 learned_addr learned my addr 172.21.15.64:0/4195735913 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:12.195 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.190+0000 7fe64ffff700 1 --2- 172.21.15.64:0/4195735913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe65015c3e0 0x7fe65015c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.195 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.191+0000 7fe64effd700 1 -- 172.21.15.64:0/4195735913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe65015c3e0 msgr2=0x7fe65015c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:12.195 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.191+0000 7fe64effd700 1 --2- 172.21.15.64:0/4195735913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe65015c3e0 0x7fe65015c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.195 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.191+0000 7fe64effd700 1 -- 172.21.15.64:0/4195735913 >> v1:172.21.15.86:6789/0 conn(0x7fe650068760 legacy=0x7fe65006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.196 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.191+0000 7fe64e7fc700 1 -- 172.21.15.64:0/4195735913 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736259621 0 0) 0x7fe6500c8350 con 0x7fe650068760 2023-06-23T00:40:12.196 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.191+0000 7fe64effd700 1 -- 172.21.15.64:0/4195735913 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6500c7a20 con 0x7fe650068b30 2023-06-23T00:40:12.196 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.191+0000 7fe64effd700 1 --2- 172.21.15.64:0/4195735913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe650068b30 0x7fe6500c87e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe6440099b0 tx=0x7fe644009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23aafe4ccfc6f54 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:12.196 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.191+0000 7fe64e7fc700 1 -- 172.21.15.64:0/4195735913 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644009ea0 con 0x7fe650068b30 2023-06-23T00:40:12.196 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.191+0000 7fe64e7fc700 1 -- 172.21.15.64:0/4195735913 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64400b860 con 0x7fe650068b30 2023-06-23T00:40:12.196 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.191+0000 7fe64e7fc700 1 -- 172.21.15.64:0/4195735913 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64400bc00 con 0x7fe650068b30 2023-06-23T00:40:12.196 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.191+0000 7fe655dfd700 1 -- 172.21.15.64:0/4195735913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe650068b30 msgr2=0x7fe6500c87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.196 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.191+0000 7fe655dfd700 1 --2- 172.21.15.64:0/4195735913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe650068b30 0x7fe6500c87e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe6440099b0 tx=0x7fe644009250 comp rx=0 tx=0).stop 2023-06-23T00:40:12.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.192+0000 7fe655dfd700 1 -- 172.21.15.64:0/4195735913 shutdown_connections 2023-06-23T00:40:12.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.192+0000 7fe655dfd700 1 --2- 172.21.15.64:0/4195735913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe65015c3e0 0x7fe65015c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.192+0000 7fe655dfd700 1 --2- 172.21.15.64:0/4195735913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe650068b30 0x7fe6500c87e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.192+0000 7fe655dfd700 1 -- 172.21.15.64:0/4195735913 >> 172.21.15.64:0/4195735913 conn(0x7fe6500c9150 msgr2=0x7fe6500c9550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:12.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.192+0000 7fe655dfd700 1 -- 172.21.15.64:0/4195735913 shutdown_connections 2023-06-23T00:40:12.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.192+0000 7fe655dfd700 1 -- 172.21.15.64:0/4195735913 wait complete. 2023-06-23T00:40:12.199 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.193+0000 7fe655dfd700 1 Processor -- start 2023-06-23T00:40:12.199 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.193+0000 7fe655dfd700 1 -- start start 2023-06-23T00:40:12.199 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.193+0000 7fe655dfd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe650068760 0x7fe6500c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.199 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.193+0000 7fe655dfd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe650068b30 0x7fe65015ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe655dfd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe65015c3e0 0x7fe650149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe655dfd700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65016cfb0 con 0x7fe65015c3e0 2023-06-23T00:40:12.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe655dfd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65016d290 con 0x7fe650068760 2023-06-23T00:40:12.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe655dfd700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65016d570 con 0x7fe650068b30 2023-06-23T00:40:12.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe65015c3e0 0x7fe650149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64effd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe650068b30 0x7fe65015ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe65015c3e0 0x7fe650149d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51702/0 (socket says 172.21.15.64:51702) 2023-06-23T00:40:12.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe650068760 0x7fe6500c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64ffff700 1 -- 172.21.15.64:0/2835479504 learned_addr learned my addr 172.21.15.64:0/2835479504 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:12.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64effd700 1 -- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe650068760 msgr2=0x7fe6500c63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64effd700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe650068760 0x7fe6500c63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64effd700 1 -- 172.21.15.64:0/2835479504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe65015c3e0 msgr2=0x7fe650149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64effd700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe65015c3e0 0x7fe650149d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64effd700 1 -- 172.21.15.64:0/2835479504 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6440094f0 con 0x7fe650068b30 2023-06-23T00:40:12.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64ffff700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe65015c3e0 0x7fe650149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:12.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe64effd700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe650068b30 0x7fe65015ca60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe650064670 tx=0x7fe644017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:12.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe6377fe700 1 -- 172.21.15.64:0/2835479504 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64400bbe0 con 0x7fe650068b30 2023-06-23T00:40:12.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe6377fe700 1 -- 172.21.15.64:0/2835479504 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64401be00 con 0x7fe650068b30 2023-06-23T00:40:12.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe655dfd700 1 -- 172.21.15.64:0/2835479504 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65016d850 con 0x7fe650068b30 2023-06-23T00:40:12.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe6377fe700 1 -- 172.21.15.64:0/2835479504 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644011630 con 0x7fe650068b30 2023-06-23T00:40:12.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.194+0000 7fe655dfd700 1 -- 172.21.15.64:0/2835479504 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6500c6900 con 0x7fe650068b30 2023-06-23T00:40:12.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.195+0000 7fe655dfd700 1 -- 172.21.15.64:0/2835479504 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6500b0730 con 0x7fe650068b30 2023-06-23T00:40:12.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.196+0000 7fe6377fe700 1 -- 172.21.15.64:0/2835479504 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7fe644011a60 con 0x7fe650068b30 2023-06-23T00:40:12.203 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.196+0000 7fe6377fe700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fe63c066e00 0x7fe63c0692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.203 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.196+0000 7fe6377fe700 1 -- 172.21.15.64:0/2835479504 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fe644085490 con 0x7fe650068b30 2023-06-23T00:40:12.203 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.196+0000 7fe64f7fe700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fe63c066e00 0x7fe63c0692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.203 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.197+0000 7fe64f7fe700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fe63c066e00 0x7fe63c0692b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe640001230 tx=0x7fe640007cd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:12.203 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.201+0000 7fe6377fe700 1 -- 172.21.15.64:0/2835479504 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fe6500b0730 con 0x7fe650068b30 2023-06-23T00:40:12.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.440+0000 7fe655dfd700 1 -- 172.21.15.64:0/2835479504 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6500b0730 con 0x7fe650068b30 2023-06-23T00:40:12.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.441+0000 7fe6377fe700 1 -- 172.21.15.64:0/2835479504 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3521 (secure 0 0 0) 0x7fe644015de0 con 0x7fe650068b30 2023-06-23T00:40:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:12.443 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6834/1840968204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6833","nonce":1840968204},{"type":"v1","addr":"172.21.15.64:6834","nonce":1840968204}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/1141461387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1141461387},{"type":"v1","addr":"172.21.15.86:6835","nonce":1141461387}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1370273364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1370273364},{"type":"v1","addr":"172.21.15.64:6835","nonce":1370273364}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/895538016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":895538016},{"type":"v1","addr":"172.21.15.86:6837","nonce":895538016}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-06-23T00:40:12.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.442+0000 7fe655dfd700 1 -- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fe63c066e00 msgr2=0x7fe63c0692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.442+0000 7fe655dfd700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fe63c066e00 0x7fe63c0692b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe640001230 tx=0x7fe640007cd0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.442+0000 7fe655dfd700 1 -- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe650068b30 msgr2=0x7fe65015ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.442+0000 7fe655dfd700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe650068b30 0x7fe65015ca60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe650064670 tx=0x7fe644017000 comp rx=0 tx=0).stop 2023-06-23T00:40:12.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.443+0000 7fe655dfd700 1 -- 172.21.15.64:0/2835479504 shutdown_connections 2023-06-23T00:40:12.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.443+0000 7fe655dfd700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fe63c066e00 0x7fe63c0692b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.443+0000 7fe655dfd700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe65015c3e0 0x7fe650149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.443+0000 7fe655dfd700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe650068760 0x7fe6500c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.443+0000 7fe655dfd700 1 --2- 172.21.15.64:0/2835479504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe650068b30 0x7fe65015ca60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.443+0000 7fe655dfd700 1 -- 172.21.15.64:0/2835479504 >> 172.21.15.64:0/2835479504 conn(0x7fe6500c9150 msgr2=0x7fe650155cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:12.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.443+0000 7fe655dfd700 1 -- 172.21.15.64:0/2835479504 shutdown_connections 2023-06-23T00:40:12.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.443+0000 7fe655dfd700 1 -- 172.21.15.64:0/2835479504 wait complete. 2023-06-23T00:40:12.447 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 3 2023-06-23T00:40:12.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T00:40:12.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.574+0000 7f0f548a4700 1 Processor -- start 2023-06-23T00:40:12.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.574+0000 7f0f548a4700 1 -- start start 2023-06-23T00:40:12.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.575+0000 7f0f548a4700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f0f500d04f0 0x7f0f5015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.575+0000 7f0f548a4700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0f500d08c0 0x7f0f500cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.575+0000 7f0f548a4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f501569b0 0x7f0f50156e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.575+0000 7f0f548a4700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5015aaa0 con 0x7f0f500d04f0 2023-06-23T00:40:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.575+0000 7f0f548a4700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f500ce5c0 con 0x7f0f500d08c0 2023-06-23T00:40:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.575+0000 7f0f548a4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f500ccf70 con 0x7f0f501569b0 2023-06-23T00:40:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.575+0000 7f0f4e59c700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f0f500d04f0 0x7f0f5015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.575+0000 7f0f4e59c700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f0f500d04f0 0x7f0f5015a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51718/0 (socket says 172.21.15.64:51718) 2023-06-23T00:40:12.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.575+0000 7f0f4e59c700 1 -- 172.21.15.64:0/3010413882 learned_addr learned my addr 172.21.15.64:0/3010413882 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:12.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.576+0000 7f0f4dd9b700 1 --2- 172.21.15.64:0/3010413882 >> v2:172.21.15.86:3300/0 conn(0x7f0f500d08c0 0x7f0f500cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.576+0000 7f0f4ed9d700 1 --2- 172.21.15.64:0/3010413882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f501569b0 0x7f0f50156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.576+0000 7f0f4e59c700 1 -- 172.21.15.64:0/3010413882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f501569b0 msgr2=0x7f0f50156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:12.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.576+0000 7f0f4e59c700 1 --2- 172.21.15.64:0/3010413882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f501569b0 0x7f0f50156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.576+0000 7f0f4e59c700 1 -- 172.21.15.64:0/3010413882 >> v2:172.21.15.86:3300/0 conn(0x7f0f500d08c0 msgr2=0x7f0f500cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:12.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.576+0000 7f0f4e59c700 1 --2- 172.21.15.64:0/3010413882 >> v2:172.21.15.86:3300/0 conn(0x7f0f500d08c0 0x7f0f500cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.576+0000 7f0f4e59c700 1 -- 172.21.15.64:0/3010413882 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f50157350 con 0x7f0f500d04f0 2023-06-23T00:40:12.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.576+0000 7f0f4e59c700 1 --2- 172.21.15.64:0/3010413882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f500d04f0 0x7f0f5015a410 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0f50089250 tx=0x7f0f48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad6ed337d85bed2 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:12.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.576+0000 7f0f4d59a700 1 -- 172.21.15.64:0/3010413882 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4800b860 con 0x7f0f500d04f0 2023-06-23T00:40:12.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.576+0000 7f0f4d59a700 1 -- 172.21.15.64:0/3010413882 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4800b9c0 con 0x7f0f500d04f0 2023-06-23T00:40:12.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.576+0000 7f0f4d59a700 1 -- 172.21.15.64:0/3010413882 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4800bd60 con 0x7f0f500d04f0 2023-06-23T00:40:12.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.577+0000 7f0f548a4700 1 -- 172.21.15.64:0/3010413882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f500d04f0 msgr2=0x7f0f5015a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.577+0000 7f0f548a4700 1 --2- 172.21.15.64:0/3010413882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f500d04f0 0x7f0f5015a410 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0f50089250 tx=0x7f0f48009250 comp rx=0 tx=0).stop 2023-06-23T00:40:12.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.577+0000 7f0f548a4700 1 -- 172.21.15.64:0/3010413882 shutdown_connections 2023-06-23T00:40:12.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.577+0000 7f0f548a4700 1 --2- 172.21.15.64:0/3010413882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f501569b0 0x7f0f50156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.577+0000 7f0f548a4700 1 --2- 172.21.15.64:0/3010413882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f500d04f0 0x7f0f5015a410 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.577+0000 7f0f548a4700 1 --2- 172.21.15.64:0/3010413882 >> v2:172.21.15.86:3300/0 conn(0x7f0f500d08c0 0x7f0f500cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.577+0000 7f0f548a4700 1 -- 172.21.15.64:0/3010413882 >> 172.21.15.64:0/3010413882 conn(0x7f0f5006a060 msgr2=0x7f0f5006a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:12.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.577+0000 7f0f548a4700 1 -- 172.21.15.64:0/3010413882 shutdown_connections 2023-06-23T00:40:12.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.577+0000 7f0f548a4700 1 -- 172.21.15.64:0/3010413882 wait complete. 2023-06-23T00:40:12.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.578+0000 7f0f548a4700 1 Processor -- start 2023-06-23T00:40:12.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.578+0000 7f0f548a4700 1 -- start start 2023-06-23T00:40:12.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.578+0000 7f0f548a4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f500d04f0 0x7f0f50153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.578+0000 7f0f548a4700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500d08c0 0x7f0f5016d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f548a4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f501569b0 0x7f0f50155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f548a4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f500b39b0 con 0x7f0f500d04f0 2023-06-23T00:40:12.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f548a4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5016d9f0 con 0x7f0f501569b0 2023-06-23T00:40:12.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f548a4700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f500c53e0 con 0x7f0f500d08c0 2023-06-23T00:40:12.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f4e59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f500d04f0 0x7f0f50153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f4e59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f500d04f0 0x7f0f50153c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51722/0 (socket says 172.21.15.64:51722) 2023-06-23T00:40:12.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f4e59c700 1 -- 172.21.15.64:0/2330779751 learned_addr learned my addr 172.21.15.64:0/2330779751 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:12.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f4ed9d700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f501569b0 0x7f0f50155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f4dd9b700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500d08c0 0x7f0f5016d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f4e59c700 1 -- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500d08c0 msgr2=0x7f0f5016d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f4e59c700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500d08c0 0x7f0f5016d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f4e59c700 1 -- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f501569b0 msgr2=0x7f0f50155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f4e59c700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f501569b0 0x7f0f50155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f4e59c700 1 -- 172.21.15.64:0/2330779751 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f480094f0 con 0x7f0f500d04f0 2023-06-23T00:40:12.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f4e59c700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f500d04f0 0x7f0f50153c40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0f50063e80 tx=0x7f0f48017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:12.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f467fc700 1 -- 172.21.15.64:0/2330779751 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f48011dc0 con 0x7f0f500d04f0 2023-06-23T00:40:12.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f467fc700 1 -- 172.21.15.64:0/2330779751 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f48024440 con 0x7f0f500d04f0 2023-06-23T00:40:12.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f548a4700 1 -- 172.21.15.64:0/2330779751 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f500b59a0 con 0x7f0f500d04f0 2023-06-23T00:40:12.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f467fc700 1 -- 172.21.15.64:0/2330779751 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f48024800 con 0x7f0f500d04f0 2023-06-23T00:40:12.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.579+0000 7f0f548a4700 1 -- 172.21.15.64:0/2330779751 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f500b5650 con 0x7f0f500d04f0 2023-06-23T00:40:12.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.580+0000 7f0f548a4700 1 -- 172.21.15.64:0/2330779751 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5004c1b0 con 0x7f0f500d04f0 2023-06-23T00:40:12.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.581+0000 7f0f467fc700 1 -- 172.21.15.64:0/2330779751 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f0f4801a420 con 0x7f0f500d04f0 2023-06-23T00:40:12.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.581+0000 7f0f467fc700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0f30066e00 0x7f0f300692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.581+0000 7f0f467fc700 1 -- 172.21.15.64:0/2330779751 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f0f48085fc0 con 0x7f0f500d04f0 2023-06-23T00:40:12.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.581+0000 7f0f4dd9b700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0f30066e00 0x7f0f300692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.582+0000 7f0f4dd9b700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0f30066e00 0x7f0f300692b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0f500b38a0 tx=0x7f0f38008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:12.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.586+0000 7f0f467fc700 1 -- 172.21.15.64:0/2330779751 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f0f480535c0 con 0x7f0f500d04f0 2023-06-23T00:40:12.677 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.674+0000 7f0f467fc700 1 -- 172.21.15.64:0/2330779751 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f0f48053990 con 0x7f0f500d04f0 2023-06-23T00:40:12.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.782+0000 7f0f548a4700 1 -- 172.21.15.64:0/2330779751 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f500b3b20 con 0x7f0f500d04f0 2023-06-23T00:40:12.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.783+0000 7f0f467fc700 1 -- 172.21.15.64:0/2330779751 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+13648 (secure 0 0 0) 0x7f0f500b3b20 con 0x7f0f500d04f0 2023-06-23T00:40:12.786 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:12.786 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":16,"fsid":"72efde62-4c91-4277-87ee-356e50bd4cf5","created":"2023-06-23T00:39:58.078232+0000","modified":"2023-06-23T00:40:11.984587+0000","last_up_change":"2023-06-23T00:40:08.853479+0000","last_in_change":"2023-06-23T00:40:00.493656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-23T00:40:09.294447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-23T00:40:09.719568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":88027},{"type":"v1","addr":"172.21.15.64:6801","nonce":88027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":88027},{"type":"v1","addr":"172.21.15.64:6803","nonce":88027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":88027},{"type":"v1","addr":"172.21.15.64:6807","nonce":88027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":88027},{"type":"v1","addr":"172.21.15.64:6805","nonce":88027}]},"public_addr":"172.21.15.64:6801/88027","cluster_addr":"172.21.15.64:6803/88027","heartbeat_back_addr":"172.21.15.64:6807/88027","heartbeat_front_addr":"172.21.15.64:6805/88027","state":["exists","up"]},{"osd":1,"uuid":"f0698c9d-c67a-42ac-8d36-838836c514d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":88062},{"type":"v1","addr":"172.21.15.64:6825","nonce":88062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":88062},{"type":"v1","addr":"172.21.15.64:6827","nonce":88062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":88062},{"type":"v1","addr":"172.21.15.64:6831","nonce":88062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":88062},{"type":"v1","addr":"172.21.15.64:6829","nonce":88062}]},"public_addr":"172.21.15.64:6825/88062","cluster_addr":"172.21.15.64:6827/88062","heartbeat_back_addr":"172.21.15.64:6831/88062","heartbeat_front_addr":"172.21.15.64:6829/88062","state":["exists","up"]},{"osd":2,"uuid":"a5ece869-9da7-40e8-8f1d-bbaf67bb2187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":88045},{"type":"v1","addr":"172.21.15.64:6817","nonce":88045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":88045},{"type":"v1","addr":"172.21.15.64:6819","nonce":88045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":88045},{"type":"v1","addr":"172.21.15.64:6823","nonce":88045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":88045},{"type":"v1","addr":"172.21.15.64:6821","nonce":88045}]},"public_addr":"172.21.15.64:6817/88045","cluster_addr":"172.21.15.64:6819/88045","heartbeat_back_addr":"172.21.15.64:6823/88045","heartbeat_front_addr":"172.21.15.64:6821/88045","state":["exists","up"]},{"osd":3,"uuid":"21067fac-663b-4ecf-ae67-bf235161638e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":88044},{"type":"v1","addr":"172.21.15.64:6809","nonce":88044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":88044},{"type":"v1","addr":"172.21.15.64:6811","nonce":88044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":88044},{"type":"v1","addr":"172.21.15.64:6815","nonce":88044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":88044},{"type":"v1","addr":"172.21.15.64:6813","nonce":88044}]},"public_addr":"172.21.15.64:6809/88044","cluster_addr":"172.21.15.64:6811/88044","heartbeat_back_addr":"172.21.15.64:6815/88044","heartbeat_front_addr":"172.21.15.64:6813/88044","state":["exists","up"]},{"osd":4,"uuid":"6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":118144},{"type":"v1","addr":"172.21.15.86:6801","nonce":118144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":118144},{"type":"v1","addr":"172.21.15.86:6803","nonce":118144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":118144},{"type":"v1","addr":"172.21.15.86:6807","nonce":118144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":118144},{"type":"v1","addr":"172.21.15.86:6805","nonce":118144}]},"public_addr":"172.21.15.86:6801/118144","cluster_addr":"172.21.15.86:6803/118144","heartbeat_back_addr":"172.21.15.86:6807/118144","heartbeat_front_addr":"172.21.15.86:6805/118144","state":["exists","up"]},{"osd":5,"uuid":"0e987a73-8b36-4f26-94fd-f0135d6062c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":118169},{"type":"v1","addr":"172.21.15.86:6817","nonce":118169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":118169},{"type":"v1","addr":"172.21.15.86:6819","nonce":118169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":118169},{"type":"v1","addr":"172.21.15.86:6823","nonce":118169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":118169},{"type":"v1","addr":"172.21.15.86:6821","nonce":118169}]},"public_addr":"172.21.15.86:6817/118169","cluster_addr":"172.21.15.86:6819/118169","heartbeat_back_addr":"172.21.15.86:6823/118169","heartbeat_front_addr":"172.21.15.86:6821/118169","state":["exists","up"]},{"osd":6,"uuid":"7e40bfd5-0f9f-448c-8ccf-2f44bf82a622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":118175},{"type":"v1","addr":"172.21.15.86:6825","nonce":118175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":118175},{"type":"v1","addr":"172.21.15.86:6827","nonce":118175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":118175},{"type":"v1","addr":"172.21.15.86:6831","nonce":118175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":118175},{"type":"v1","addr":"172.21.15.86:6829","nonce":118175}]},"public_addr":"172.21.15.86:6825/118175","cluster_addr":"172.21.15.86:6827/118175","heartbeat_back_addr":"172.21.15.86:6831/118175","heartbeat_front_addr":"172.21.15.86:6829/118175","state":["exists","up"]},{"osd":7,"uuid":"2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":118145},{"type":"v1","addr":"172.21.15.86:6809","nonce":118145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":118145},{"type":"v1","addr":"172.21.15.86:6811","nonce":118145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":118145},{"type":"v1","addr":"172.21.15.86:6815","nonce":118145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":118145},{"type":"v1","addr":"172.21.15.86:6813","nonce":118145}]},"public_addr":"172.21.15.86:6809/118145","cluster_addr":"172.21.15.86:6811/118145","heartbeat_back_addr":"172.21.15.86:6815/118145","heartbeat_front_addr":"172.21.15.86:6813/118145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.319777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.692469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.685180+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.648752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.461468+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.727692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.701726+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.703200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T00:40:12.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 -- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0f30066e00 msgr2=0x7f0f300692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0f30066e00 0x7f0f300692b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0f500b38a0 tx=0x7f0f38008040 comp rx=0 tx=0).stop 2023-06-23T00:40:12.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 -- 172.21.15.64:0/2330779751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f500d04f0 msgr2=0x7f0f50153c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f500d04f0 0x7f0f50153c40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0f50063e80 tx=0x7f0f48017000 comp rx=0 tx=0).stop 2023-06-23T00:40:12.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 -- 172.21.15.64:0/2330779751 shutdown_connections 2023-06-23T00:40:12.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f0f30066e00 0x7f0f300692b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f501569b0 0x7f0f50155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f500d04f0 0x7f0f50153c40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 --2- 172.21.15.64:0/2330779751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500d08c0 0x7f0f5016d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 -- 172.21.15.64:0/2330779751 >> 172.21.15.64:0/2330779751 conn(0x7f0f5006a060 msgr2=0x7f0f50158b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:12.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 -- 172.21.15.64:0/2330779751 shutdown_connections 2023-06-23T00:40:12.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.785+0000 7f0f548a4700 1 -- 172.21.15.64:0/2330779751 wait complete. 2023-06-23T00:40:12.799 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-06-23T00:40:09.294447+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-06-23T00:40:09.719568+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-23T00:40:12.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-23T00:40:12.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.920+0000 7f4e821f7700 1 Processor -- start 2023-06-23T00:40:12.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.920+0000 7f4e821f7700 1 -- start start 2023-06-23T00:40:12.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.921+0000 7f4e821f7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7c0d04f0 0x7f4e7c15a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.921+0000 7f4e821f7700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4e7c0d08c0 0x7f4e7c0cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.921+0000 7f4e821f7700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7c15aaa0 con 0x7f4e7c1569b0 2023-06-23T00:40:12.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.921+0000 7f4e821f7700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e7c0ce5c0 con 0x7f4e7c0d08c0 2023-06-23T00:40:12.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.921+0000 7f4e821f7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0ccf70 con 0x7f4e7c0d04f0 2023-06-23T00:40:12.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.921+0000 7f4e7bfff700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4e7c1569b0 0x7f4e7c15cf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36638/0 (socket says 172.21.15.64:36638) 2023-06-23T00:40:12.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.921+0000 7f4e7bfff700 1 -- 172.21.15.64:0/3364806443 learned_addr learned my addr 172.21.15.64:0/3364806443 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:12.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7affd700 1 --2- 172.21.15.64:0/3364806443 >> v2:172.21.15.86:3300/0 conn(0x7f4e7c0d08c0 0x7f4e7c0cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7b7fe700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7c0d04f0 0x7f4e7c15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212034406 0 0) 0x7f4e7c15aaa0 con 0x7f4e7c1569b0 2023-06-23T00:40:12.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e64003570 con 0x7f4e7c1569b0 2023-06-23T00:40:12.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3592804134 0 0) 0x7f4e64003570 con 0x7f4e7c1569b0 2023-06-23T00:40:12.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7c0d04f0 msgr2=0x7f4e7c15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:12.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7c0d04f0 0x7f4e7c15a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 >> v2:172.21.15.86:3300/0 conn(0x7f4e7c0d08c0 msgr2=0x7f4e7c0cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:12.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 --2- 172.21.15.64:0/3364806443 >> v2:172.21.15.86:3300/0 conn(0x7f4e7c0d08c0 0x7f4e7c0cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c0cd1d0 con 0x7f4e7c1569b0 2023-06-23T00:40:12.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 939283155 0 0) 0x7f4e740017f0 con 0x7f4e7c1569b0 2023-06-23T00:40:12.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 >> v1:172.21.15.64:6789/0 conn(0x7f4e7c1569b0 legacy=0x7f4e7c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e64003a60 0x7f4e64003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7c0d04f0 0x7f4e64004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e64004d50 0x7f4e640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0ce5c0 con 0x7f4e64003a60 2023-06-23T00:40:12.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0ccf70 con 0x7f4e64004d50 2023-06-23T00:40:12.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6400ba60 con 0x7f4e7c0d04f0 2023-06-23T00:40:12.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7b7fe700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e64003a60 0x7f4e64003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7affd700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7c0d04f0 0x7f4e64004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7bfff700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e64004d50 0x7f4e640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7b7fe700 1 -- 172.21.15.64:0/3364806443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7c0d04f0 msgr2=0x7f4e64004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:12.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7b7fe700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e7c0d04f0 0x7f4e64004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7b7fe700 1 -- 172.21.15.64:0/3364806443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e64004d50 msgr2=0x7f4e640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:12.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7b7fe700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e64004d50 0x7f4e640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7b7fe700 1 -- 172.21.15.64:0/3364806443 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e64003570 con 0x7f4e64003a60 2023-06-23T00:40:12.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.922+0000 7f4e7b7fe700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e64003a60 0x7f4e64003e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009c00 tx=0x7f4e6c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3a3ae568fbdade server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:12.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.923+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c00e6b0 con 0x7f4e64003a60 2023-06-23T00:40:12.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.923+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6c00e810 con 0x7f4e64003a60 2023-06-23T00:40:12.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.923+0000 7f4e7a7fc700 1 -- 172.21.15.64:0/3364806443 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c00ebd0 con 0x7f4e64003a60 2023-06-23T00:40:12.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.923+0000 7f4e821f7700 1 -- 172.21.15.64:0/3364806443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e64003a60 msgr2=0x7f4e64003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.923+0000 7f4e821f7700 1 --2- 172.21.15.64:0/3364806443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e64003a60 0x7f4e64003e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009c00 tx=0x7f4e6c0091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.923+0000 7f4e7bfff700 1 -- 172.21.15.64:0/3364806443 reap_dead start 2023-06-23T00:40:12.932 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.923+0000 7f4e821f7700 1 -- 172.21.15.64:0/3364806443 shutdown_connections 2023-06-23T00:40:12.932 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.923+0000 7f4e821f7700 1 -- 172.21.15.64:0/3364806443 >> 172.21.15.64:0/3364806443 conn(0x7f4e7c06a060 msgr2=0x7f4e7c06a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:12.932 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.923+0000 7f4e821f7700 1 -- 172.21.15.64:0/3364806443 shutdown_connections 2023-06-23T00:40:12.932 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.923+0000 7f4e821f7700 1 -- 172.21.15.64:0/3364806443 wait complete. 2023-06-23T00:40:12.934 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.924+0000 7f4e821f7700 1 Processor -- start 2023-06-23T00:40:12.934 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.924+0000 7f4e821f7700 1 -- start start 2023-06-23T00:40:12.934 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e821f7700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e64003a60 0x7f4e7c152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.935 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e821f7700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7c15b5c0 0x7f4e7c0b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.935 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e821f7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7c0b53b0 0x7f4e7c0b5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.935 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e821f7700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c0b0100 con 0x7f4e7c15b5c0 2023-06-23T00:40:12.935 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e821f7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c152d50 con 0x7f4e7c0b53b0 2023-06-23T00:40:12.935 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e821f7700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7c15c610 con 0x7f4e64003a60 2023-06-23T00:40:12.935 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e7affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7c15b5c0 0x7f4e7c0b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.935 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e7affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7c15b5c0 0x7f4e7c0b4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51738/0 (socket says 172.21.15.64:51738) 2023-06-23T00:40:12.936 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e7affd700 1 -- 172.21.15.64:0/376411294 learned_addr learned my addr 172.21.15.64:0/376411294 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:12.936 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e7b7fe700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e64003a60 0x7f4e7c152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.936 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e7bfff700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7c0b53b0 0x7f4e7c0b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.936 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e7affd700 1 -- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e64003a60 msgr2=0x7f4e7c152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e7affd700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e64003a60 0x7f4e7c152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.925+0000 7f4e7affd700 1 -- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7c0b53b0 msgr2=0x7f4e7c0b5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:12.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.926+0000 7f4e7affd700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7c0b53b0 0x7f4e7c0b5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:12.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.926+0000 7f4e7affd700 1 -- 172.21.15.64:0/376411294 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7c15bd70 con 0x7f4e7c15b5c0 2023-06-23T00:40:12.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.926+0000 7f4e7bfff700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7c0b53b0 0x7f4e7c0b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T00:40:12.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.926+0000 7f4e7affd700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7c15b5c0 0x7f4e7c0b4e70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4e70011d70 tx=0x7f4e700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:12.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.926+0000 7f4e637fe700 1 -- 172.21.15.64:0/376411294 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000fac0 con 0x7f4e7c15b5c0 2023-06-23T00:40:12.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.926+0000 7f4e637fe700 1 -- 172.21.15.64:0/376411294 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7000fc20 con 0x7f4e7c15b5c0 2023-06-23T00:40:12.939 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.926+0000 7f4e821f7700 1 -- 172.21.15.64:0/376411294 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7c15c050 con 0x7f4e7c15b5c0 2023-06-23T00:40:12.939 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.926+0000 7f4e637fe700 1 -- 172.21.15.64:0/376411294 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70014440 con 0x7f4e7c15b5c0 2023-06-23T00:40:12.939 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.926+0000 7f4e821f7700 1 -- 172.21.15.64:0/376411294 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7c15ba30 con 0x7f4e7c15b5c0 2023-06-23T00:40:12.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.927+0000 7f4e821f7700 1 -- 172.21.15.64:0/376411294 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e7c0b0270 con 0x7f4e7c15b5c0 2023-06-23T00:40:12.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.927+0000 7f4e637fe700 1 -- 172.21.15.64:0/376411294 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f4e70014870 con 0x7f4e7c15b5c0 2023-06-23T00:40:12.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.928+0000 7f4e637fe700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4e68066e90 0x7f4e68069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:12.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.928+0000 7f4e637fe700 1 -- 172.21.15.64:0/376411294 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f4e70018030 con 0x7f4e7c15b5c0 2023-06-23T00:40:12.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.928+0000 7f4e7b7fe700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4e68066e90 0x7f4e68069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:12.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.928+0000 7f4e7b7fe700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4e68066e90 0x7f4e68069340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0091e0 tx=0x7f4e6c01a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:12.942 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:12.932+0000 7f4e637fe700 1 -- 172.21.15.64:0/376411294 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f4e7c0b0270 con 0x7f4e7c15b5c0 2023-06-23T00:40:13.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.128+0000 7f4e821f7700 1 -- 172.21.15.64:0/376411294 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4e7c0b0240 con 0x7f4e7c15b5c0 2023-06-23T00:40:13.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.128+0000 7f4e637fe700 1 -- 172.21.15.64:0/376411294 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4e70022020 con 0x7f4e7c15b5c0 2023-06-23T00:40:13.131 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 8 2023-06-23T00:40:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 -- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4e68066e90 msgr2=0x7f4e68069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4e68066e90 0x7f4e68069340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0091e0 tx=0x7f4e6c01a000 comp rx=0 tx=0).stop 2023-06-23T00:40:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 -- 172.21.15.64:0/376411294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7c15b5c0 msgr2=0x7f4e7c0b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7c15b5c0 0x7f4e7c0b4e70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4e70011d70 tx=0x7f4e700091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 -- 172.21.15.64:0/376411294 shutdown_connections 2023-06-23T00:40:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4e68066e90 0x7f4e68069340 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e7c0b53b0 0x7f4e7c0b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e64003a60 0x7f4e7c152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 --2- 172.21.15.64:0/376411294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e7c15b5c0 0x7f4e7c0b4e70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 -- 172.21.15.64:0/376411294 >> 172.21.15.64:0/376411294 conn(0x7f4e7c06a060 msgr2=0x7f4e7c157770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:13.134 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 -- 172.21.15.64:0/376411294 shutdown_connections 2023-06-23T00:40:13.135 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.130+0000 7f4e821f7700 1 -- 172.21.15.64:0/376411294 wait complete. 2023-06-23T00:40:13.145 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T00:40:13.266 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.263+0000 7f9f1430f700 1 Processor -- start 2023-06-23T00:40:13.266 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.263+0000 7f9f1430f700 1 -- start start 2023-06-23T00:40:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.264+0000 7f9f1430f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c0c7430 0x7f9f0c162130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.269 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.264+0000 7f9f1430f700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c15bb30 con 0x7f9f0c0c82e0 2023-06-23T00:40:13.270 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.264+0000 7f9f1430f700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c069e40 con 0x7f9f0c157000 2023-06-23T00:40:13.270 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.264+0000 7f9f1430f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c15af20 con 0x7f9f0c0c7430 2023-06-23T00:40:13.270 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.264+0000 7f9f118aa700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f9f0c0c82e0 0x7f9f0c0c6d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36644/0 (socket says 172.21.15.64:36644) 2023-06-23T00:40:13.270 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.264+0000 7f9f118aa700 1 -- 172.21.15.64:0/1221092654 learned_addr learned my addr 172.21.15.64:0/1221092654 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:13.270 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.264+0000 7f9f128ac700 1 --2- 172.21.15.64:0/1221092654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c0c7430 0x7f9f0c162130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.271 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.265+0000 7f9f110a9700 1 -- 172.21.15.64:0/1221092654 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200271527 0 0) 0x7f9f0c15bb30 con 0x7f9f0c0c82e0 2023-06-23T00:40:13.271 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.265+0000 7f9f110a9700 1 -- 172.21.15.64:0/1221092654 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef4003570 con 0x7f9f0c0c82e0 2023-06-23T00:40:13.271 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.265+0000 7f9f110a9700 1 -- 172.21.15.64:0/1221092654 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633976971 0 0) 0x7f9f0c069e40 con 0x7f9f0c157000 2023-06-23T00:40:13.272 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.265+0000 7f9f110a9700 1 -- 172.21.15.64:0/1221092654 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0c15bb30 con 0x7f9f0c157000 2023-06-23T00:40:13.272 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.265+0000 7f9f128ac700 1 -- 172.21.15.64:0/1221092654 >> v1:172.21.15.86:6789/0 conn(0x7f9f0c157000 legacy=0x7f9f0c069730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.272 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.265+0000 7f9f128ac700 1 -- 172.21.15.64:0/1221092654 >> v1:172.21.15.64:6789/0 conn(0x7f9f0c0c82e0 legacy=0x7f9f0c0c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.272 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.265+0000 7f9f128ac700 1 -- 172.21.15.64:0/1221092654 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c15b200 con 0x7f9f0c0c7430 2023-06-23T00:40:13.273 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.265+0000 7f9f128ac700 1 --2- 172.21.15.64:0/1221092654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c0c7430 0x7f9f0c162130 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009e90 tx=0x7f9efc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cf644e5c473b6f9 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:13.273 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.265+0000 7f9f110a9700 1 -- 172.21.15.64:0/1221092654 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc00e070 con 0x7f9f0c0c7430 2023-06-23T00:40:13.273 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.265+0000 7f9f110a9700 1 -- 172.21.15.64:0/1221092654 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9efc00bb80 con 0x7f9f0c0c7430 2023-06-23T00:40:13.273 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.265+0000 7f9f110a9700 1 -- 172.21.15.64:0/1221092654 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9efc012430 con 0x7f9f0c0c7430 2023-06-23T00:40:13.274 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.266+0000 7f9f1430f700 1 -- 172.21.15.64:0/1221092654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c0c7430 msgr2=0x7f9f0c162130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.274 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.266+0000 7f9f1430f700 1 --2- 172.21.15.64:0/1221092654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c0c7430 0x7f9f0c162130 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009e90 tx=0x7f9efc009780 comp rx=0 tx=0).stop 2023-06-23T00:40:13.274 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.266+0000 7f9f1430f700 1 -- 172.21.15.64:0/1221092654 shutdown_connections 2023-06-23T00:40:13.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.266+0000 7f9f1430f700 1 --2- 172.21.15.64:0/1221092654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c0c7430 0x7f9f0c162130 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.266+0000 7f9f1430f700 1 -- 172.21.15.64:0/1221092654 >> 172.21.15.64:0/1221092654 conn(0x7f9f0c0c9150 msgr2=0x7f9f0c0c9550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:13.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.266+0000 7f9f1430f700 1 -- 172.21.15.64:0/1221092654 shutdown_connections 2023-06-23T00:40:13.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.266+0000 7f9f1430f700 1 -- 172.21.15.64:0/1221092654 wait complete. 2023-06-23T00:40:13.276 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.267+0000 7f9f1430f700 1 Processor -- start 2023-06-23T00:40:13.276 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.267+0000 7f9f1430f700 1 -- start start 2023-06-23T00:40:13.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f1430f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f0c0c7430 0x7f9f0c159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f1430f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0c82e0 0x7f9f0c14c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f1430f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c157000 0x7f9f0c1573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f1430f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c159320 con 0x7f9f0c0c7430 2023-06-23T00:40:13.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f1430f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c1586e0 con 0x7f9f0c0c82e0 2023-06-23T00:40:13.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f1430f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c157ad0 con 0x7f9f0c157000 2023-06-23T00:40:13.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f120ab700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f0c0c7430 0x7f9f0c159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f120ab700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f0c0c7430 0x7f9f0c159e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51742/0 (socket says 172.21.15.64:51742) 2023-06-23T00:40:13.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f120ab700 1 -- 172.21.15.64:0/1220203284 learned_addr learned my addr 172.21.15.64:0/1220203284 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:13.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f128ac700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c157000 0x7f9f0c1573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f118aa700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0c82e0 0x7f9f0c14c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f120ab700 1 -- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c157000 msgr2=0x7f9f0c1573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f120ab700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c157000 0x7f9f0c1573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f120ab700 1 -- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0c82e0 msgr2=0x7f9f0c14c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f120ab700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0c82e0 0x7f9f0c14c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f120ab700 1 -- 172.21.15.64:0/1220203284 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc009a20 con 0x7f9f0c0c7430 2023-06-23T00:40:13.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f118aa700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0c82e0 0x7f9f0c14c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:13.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f128ac700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c157000 0x7f9f0c1573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:13.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f120ab700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f0c0c7430 0x7f9f0c159e60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400c590 tx=0x7f9f04007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:13.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f0a7fc700 1 -- 172.21.15.64:0/1220203284 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f04010510 con 0x7f9f0c0c7430 2023-06-23T00:40:13.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f0a7fc700 1 -- 172.21.15.64:0/1220203284 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f04010670 con 0x7f9f0c0c7430 2023-06-23T00:40:13.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f1430f700 1 -- 172.21.15.64:0/1220203284 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0c157db0 con 0x7f9f0c0c7430 2023-06-23T00:40:13.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.268+0000 7f9f0a7fc700 1 -- 172.21.15.64:0/1220203284 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f04010a30 con 0x7f9f0c0c7430 2023-06-23T00:40:13.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.269+0000 7f9f1430f700 1 -- 172.21.15.64:0/1220203284 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0c158370 con 0x7f9f0c0c7430 2023-06-23T00:40:13.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.269+0000 7f9f1430f700 1 -- 172.21.15.64:0/1220203284 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f0c0b0b00 con 0x7f9f0c0c7430 2023-06-23T00:40:13.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.270+0000 7f9f0a7fc700 1 -- 172.21.15.64:0/1220203284 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f9f04002050 con 0x7f9f0c0c7430 2023-06-23T00:40:13.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.270+0000 7f9f0a7fc700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9ef8066e90 0x7f9ef8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.270+0000 7f9f0a7fc700 1 -- 172.21.15.64:0/1220203284 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f9f04083ab0 con 0x7f9f0c0c7430 2023-06-23T00:40:13.283 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.270+0000 7f9f118aa700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9ef8066e90 0x7f9ef8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.283 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.271+0000 7f9f118aa700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9ef8066e90 0x7f9ef8069340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c0ae9f0 tx=0x7f9f00009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:13.283 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.274+0000 7f9f0a7fc700 1 -- 172.21.15.64:0/1220203284 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f9f04050fd0 con 0x7f9f0c0c7430 2023-06-23T00:40:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.466+0000 7f9f1430f700 1 -- 172.21.15.64:0/1220203284 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f0c04c1b0 con 0x7f9f0c0c7430 2023-06-23T00:40:13.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.467+0000 7f9f0a7fc700 1 -- 172.21.15.64:0/1220203284 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f040545f0 con 0x7f9f0c0c7430 2023-06-23T00:40:13.470 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-06-23T00:40:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 -- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9ef8066e90 msgr2=0x7f9ef8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9ef8066e90 0x7f9ef8069340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c0ae9f0 tx=0x7f9f00009040 comp rx=0 tx=0).stop 2023-06-23T00:40:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 -- 172.21.15.64:0/1220203284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f0c0c7430 msgr2=0x7f9f0c159e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f0c0c7430 0x7f9f0c159e60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400c590 tx=0x7f9f04007e00 comp rx=0 tx=0).stop 2023-06-23T00:40:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 -- 172.21.15.64:0/1220203284 shutdown_connections 2023-06-23T00:40:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9ef8066e90 0x7f9ef8069340 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f0c157000 0x7f9f0c1573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9f0c0c7430 0x7f9f0c159e60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 --2- 172.21.15.64:0/1220203284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f0c0c82e0 0x7f9f0c14c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 -- 172.21.15.64:0/1220203284 >> 172.21.15.64:0/1220203284 conn(0x7f9f0c0c9150 msgr2=0x7f9f0c150810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 -- 172.21.15.64:0/1220203284 shutdown_connections 2023-06-23T00:40:13.473 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.469+0000 7f9f1430f700 1 -- 172.21.15.64:0/1220203284 wait complete. 2023-06-23T00:40:13.483 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-23T00:40:13.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.599+0000 7ff7b5dcf700 1 Processor -- start 2023-06-23T00:40:13.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.599+0000 7ff7b5dcf700 1 -- start start 2023-06-23T00:40:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.600+0000 7ff7b5dcf700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff7b00afa70 0x7ff7b00b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.600+0000 7ff7b5dcf700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b00aff70 con 0x7ff7b00b10a0 2023-06-23T00:40:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.600+0000 7ff7b5dcf700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b00b0250 con 0x7ff7b00b23e0 2023-06-23T00:40:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.600+0000 7ff7b5dcf700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b00b0530 con 0x7ff7b00afa70 2023-06-23T00:40:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.600+0000 7ff7af7fe700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7ff7b00b10a0 0x7ff7b015a540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36654/0 (socket says 172.21.15.64:36654) 2023-06-23T00:40:13.606 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.600+0000 7ff7af7fe700 1 -- 172.21.15.64:0/1455852100 learned_addr learned my addr 172.21.15.64:0/1455852100 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:13.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7aeffd700 1 --2- 172.21.15.64:0/1455852100 >> v2:172.21.15.86:3300/0 conn(0x7ff7b00afa70 0x7ff7b00b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160413594 0 0) 0x7ff7b00aff70 con 0x7ff7b00b10a0 2023-06-23T00:40:13.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff798003570 con 0x7ff7b00b10a0 2023-06-23T00:40:13.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519472527 0 0) 0x7ff7b00b0250 con 0x7ff7b00b23e0 2023-06-23T00:40:13.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b00aff70 con 0x7ff7b00b23e0 2023-06-23T00:40:13.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3323997857 0 0) 0x7ff798003570 con 0x7ff7b00b10a0 2023-06-23T00:40:13.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 >> v2:172.21.15.86:3300/0 conn(0x7ff7b00afa70 msgr2=0x7ff7b00b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:13.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 --2- 172.21.15.64:0/1455852100 >> v2:172.21.15.86:3300/0 conn(0x7ff7b00afa70 0x7ff7b00b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 >> v1:172.21.15.86:6789/0 conn(0x7ff7b00b23e0 legacy=0x7ff7b0098190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b00b0810 con 0x7ff7b00b10a0 2023-06-23T00:40:13.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 939283155 0 0) 0x7ff7a00015f0 con 0x7ff7b00b10a0 2023-06-23T00:40:13.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7aeffd700 1 --2- 172.21.15.64:0/1455852100 >> v2:172.21.15.86:3300/0 conn(0x7ff7b00afa70 0x7ff7b00b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:13.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 >> v1:172.21.15.64:6789/0 conn(0x7ff7b00b10a0 legacy=0x7ff7b015a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.608 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 --2- 172.21.15.64:0/1455852100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff798003bf0 0x7ff798003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 --2- 172.21.15.64:0/1455852100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff798004810 0x7ff798004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 --2- 172.21.15.64:0/1455852100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7980051c0 0x7ff79800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b00b0530 con 0x7ff7980051c0 2023-06-23T00:40:13.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79800c0b0 con 0x7ff798004810 2023-06-23T00:40:13.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79800c270 con 0x7ff798003bf0 2023-06-23T00:40:13.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7affff700 1 --2- 172.21.15.64:0/1455852100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7980051c0 0x7ff79800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.609 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7af7fe700 1 --2- 172.21.15.64:0/1455852100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff798003bf0 0x7ff798003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.610 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7aeffd700 1 --2- 172.21.15.64:0/1455852100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff798004810 0x7ff798004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.610 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7affff700 1 -- 172.21.15.64:0/1455852100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff798003bf0 msgr2=0x7ff798003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:13.610 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7affff700 1 --2- 172.21.15.64:0/1455852100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff798003bf0 0x7ff798003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.610 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7affff700 1 -- 172.21.15.64:0/1455852100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff798004810 msgr2=0x7ff798004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:13.610 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7affff700 1 --2- 172.21.15.64:0/1455852100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff798004810 0x7ff798004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.610 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7affff700 1 -- 172.21.15.64:0/1455852100 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff798003570 con 0x7ff7980051c0 2023-06-23T00:40:13.610 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.601+0000 7ff7affff700 1 -- 172.21.15.64:0/1455852100 reap_dead start 2023-06-23T00:40:13.611 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.602+0000 7ff7affff700 1 --2- 172.21.15.64:0/1455852100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7980051c0 0x7ff79800ba60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8002160 tx=0x7ff7a80017e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a8fcea5e4883dc server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:13.611 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.602+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8001af0 con 0x7ff7980051c0 2023-06-23T00:40:13.611 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.602+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a800c980 con 0x7ff7980051c0 2023-06-23T00:40:13.611 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.602+0000 7ff7ae7fc700 1 -- 172.21.15.64:0/1455852100 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a800cd00 con 0x7ff7980051c0 2023-06-23T00:40:13.611 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.602+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/1455852100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7980051c0 msgr2=0x7ff79800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.611 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.602+0000 7ff7b5dcf700 1 --2- 172.21.15.64:0/1455852100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7980051c0 0x7ff79800ba60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8002160 tx=0x7ff7a80017e0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.611 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.602+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/1455852100 shutdown_connections 2023-06-23T00:40:13.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.602+0000 7ff7b5dcf700 1 --2- 172.21.15.64:0/1455852100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7980051c0 0x7ff79800ba60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.602+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/1455852100 >> 172.21.15.64:0/1455852100 conn(0x7ff7b0068cf0 msgr2=0x7ff7b00ce090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:13.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.602+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/1455852100 shutdown_connections 2023-06-23T00:40:13.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.602+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/1455852100 wait complete. 2023-06-23T00:40:13.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.603+0000 7ff7b5dcf700 1 Processor -- start 2023-06-23T00:40:13.613 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7b5dcf700 1 -- start start 2023-06-23T00:40:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7b5dcf700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7980051c0 0x7ff7b01574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7b5dcf700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7b009b120 0x7ff7b009b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7b5dcf700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7b0167250 0x7ff7b01676c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7b5dcf700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b009bad0 con 0x7ff7b0167250 2023-06-23T00:40:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7b5dcf700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b00a8e00 con 0x7ff7980051c0 2023-06-23T00:40:13.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7b5dcf700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b0168340 con 0x7ff7b009b120 2023-06-23T00:40:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7affff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7b0167250 0x7ff7b01676c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7affff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7b0167250 0x7ff7b01676c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51766/0 (socket says 172.21.15.64:51766) 2023-06-23T00:40:13.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7affff700 1 -- 172.21.15.64:0/563733916 learned_addr learned my addr 172.21.15.64:0/563733916 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7af7fe700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7980051c0 0x7ff7b01574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.604+0000 7ff7aeffd700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7b009b120 0x7ff7b009b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.605+0000 7ff7affff700 1 -- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7b009b120 msgr2=0x7ff7b009b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.605+0000 7ff7affff700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7b009b120 0x7ff7b009b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.605+0000 7ff7affff700 1 -- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7980051c0 msgr2=0x7ff7b01574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.605+0000 7ff7affff700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7980051c0 0x7ff7b01574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.605+0000 7ff7affff700 1 -- 172.21.15.64:0/563733916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b0167fc0 con 0x7ff7b0167250 2023-06-23T00:40:13.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.605+0000 7ff7affff700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7b0167250 0x7ff7b01676c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff7b009bf10 tx=0x7ff7a8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.605+0000 7ff7977fe700 1 -- 172.21.15.64:0/563733916 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8001af0 con 0x7ff7b0167250 2023-06-23T00:40:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.605+0000 7ff7977fe700 1 -- 172.21.15.64:0/563733916 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a800c6d0 con 0x7ff7b0167250 2023-06-23T00:40:13.622 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.605+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/563733916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b00a9050 con 0x7ff7b0167250 2023-06-23T00:40:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.605+0000 7ff7977fe700 1 -- 172.21.15.64:0/563733916 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8012e40 con 0x7ff7b0167250 2023-06-23T00:40:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.605+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/563733916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b00a9610 con 0x7ff7b0167250 2023-06-23T00:40:13.623 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.606+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/563733916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b00a97d0 con 0x7ff7b0167250 2023-06-23T00:40:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.606+0000 7ff7977fe700 1 -- 172.21.15.64:0/563733916 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7ff7a8003030 con 0x7ff7b0167250 2023-06-23T00:40:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.607+0000 7ff7977fe700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7ff79c066ec0 0x7ff79c069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.607+0000 7ff7977fe700 1 -- 172.21.15.64:0/563733916 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7ff7a8085ef0 con 0x7ff7b0167250 2023-06-23T00:40:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.607+0000 7ff7af7fe700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7ff79c066ec0 0x7ff79c069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.607+0000 7ff7af7fe700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7ff79c066ec0 0x7ff79c069370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0000e10 tx=0x7ff7a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:13.624 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.611+0000 7ff7977fe700 1 -- 172.21.15.64:0/563733916 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7ff7a8053370 con 0x7ff7b0167250 2023-06-23T00:40:13.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.809+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/563733916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff7b00a97a0 con 0x7ff7b0167250 2023-06-23T00:40:13.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.809+0000 7ff7977fe700 1 -- 172.21.15.64:0/563733916 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+126 (secure 0 0 0) 0x7ff7a8056990 con 0x7ff7b0167250 2023-06-23T00:40:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:13.816 INFO:teuthology.orchestra.run.smithi064.stdout:[ 2023-06-23T00:40:13.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-06-23T00:40:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "poolnum": 1, 2023-06-23T00:40:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: "poolname": "rbd" 2023-06-23T00:40:13.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-06-23T00:40:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-06-23T00:40:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "poolnum": 2, 2023-06-23T00:40:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: "poolname": ".mgr" 2023-06-23T00:40:13.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-06-23T00:40:13.819 INFO:teuthology.orchestra.run.smithi064.stdout:] 2023-06-23T00:40:13.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.811+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7ff79c066ec0 msgr2=0x7ff79c069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.811+0000 7ff7b5dcf700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7ff79c066ec0 0x7ff79c069370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0000e10 tx=0x7ff7a0009040 comp rx=0 tx=0).stop 2023-06-23T00:40:13.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.811+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/563733916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7b0167250 msgr2=0x7ff7b01676c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.811+0000 7ff7b5dcf700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7b0167250 0x7ff7b01676c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff7b009bf10 tx=0x7ff7a8016000 comp rx=0 tx=0).stop 2023-06-23T00:40:13.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.812+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/563733916 shutdown_connections 2023-06-23T00:40:13.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.812+0000 7ff7b5dcf700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7ff79c066ec0 0x7ff79c069370 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.812+0000 7ff7b5dcf700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7b0167250 0x7ff7b01676c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.812+0000 7ff7b5dcf700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7980051c0 0x7ff7b01574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.812+0000 7ff7b5dcf700 1 --2- 172.21.15.64:0/563733916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7b009b120 0x7ff7b009b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.812+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/563733916 >> 172.21.15.64:0/563733916 conn(0x7ff7b0068cf0 msgr2=0x7ff7b00af260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:13.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.812+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/563733916 shutdown_connections 2023-06-23T00:40:13.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.812+0000 7ff7b5dcf700 1 -- 172.21.15.64:0/563733916 wait complete. 2023-06-23T00:40:13.826 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-23T00:40:13.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-23T00:40:13.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.946+0000 7f326e4c3700 1 Processor -- start 2023-06-23T00:40:13.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.946+0000 7f326e4c3700 1 -- start start 2023-06-23T00:40:13.951 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.947+0000 7f326e4c3700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3268140c90 0x7f3268068510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.951 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.947+0000 7f326e4c3700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3268068a50 0x7f32680c9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.947+0000 7f326e4c3700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326806c840 con 0x7f32681408c0 2023-06-23T00:40:13.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.947+0000 7f326e4c3700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326806ce00 con 0x7f3268068a50 2023-06-23T00:40:13.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.947+0000 7f326e4c3700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3268068e70 con 0x7f3268140c90 2023-06-23T00:40:13.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.947+0000 7f326ca60700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f3268068a50 0x7f32680c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.947+0000 7f3267fff700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f32681408c0 0x7f326806d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36668/0 (socket says 172.21.15.64:36668) 2023-06-23T00:40:13.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.947+0000 7f3267fff700 1 -- 172.21.15.64:0/4039127923 learned_addr learned my addr 172.21.15.64:0/4039127923 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:13.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.947+0000 7f32677fe700 1 --2- 172.21.15.64:0/4039127923 >> v2:172.21.15.86:3300/0 conn(0x7f3268140c90 0x7f3268068510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.948+0000 7f326ca60700 1 -- 172.21.15.64:0/4039127923 >> v2:172.21.15.86:3300/0 conn(0x7f3268140c90 msgr2=0x7f3268068510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:13.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.948+0000 7f326ca60700 1 --2- 172.21.15.64:0/4039127923 >> v2:172.21.15.86:3300/0 conn(0x7f3268140c90 0x7f3268068510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.948+0000 7f326ca60700 1 -- 172.21.15.64:0/4039127923 >> v1:172.21.15.64:6789/0 conn(0x7f32681408c0 legacy=0x7f326806d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.948+0000 7f326ca60700 1 -- 172.21.15.64:0/4039127923 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326806d9a0 con 0x7f3268068a50 2023-06-23T00:40:13.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.948+0000 7f3266ffd700 1 -- 172.21.15.64:0/4039127923 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724241503 0 0) 0x7f326806c840 con 0x7f32681408c0 2023-06-23T00:40:13.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.948+0000 7f326ca60700 1 --2- 172.21.15.64:0/4039127923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268068a50 0x7f32680c9690 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f32680646f0 tx=0x7f3260009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431c54c8e59466e4 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:13.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.948+0000 7f3266ffd700 1 -- 172.21.15.64:0/4039127923 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326000e070 con 0x7f3268068a50 2023-06-23T00:40:13.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.948+0000 7f3266ffd700 1 -- 172.21.15.64:0/4039127923 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326000bb60 con 0x7f3268068a50 2023-06-23T00:40:13.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.948+0000 7f3266ffd700 1 -- 172.21.15.64:0/4039127923 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260012430 con 0x7f3268068a50 2023-06-23T00:40:13.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.948+0000 7f326e4c3700 1 -- 172.21.15.64:0/4039127923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268068a50 msgr2=0x7f32680c9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.948+0000 7f326e4c3700 1 --2- 172.21.15.64:0/4039127923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268068a50 0x7f32680c9690 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f32680646f0 tx=0x7f3260009780 comp rx=0 tx=0).stop 2023-06-23T00:40:13.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.949+0000 7f326e4c3700 1 -- 172.21.15.64:0/4039127923 shutdown_connections 2023-06-23T00:40:13.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.949+0000 7f326e4c3700 1 --2- 172.21.15.64:0/4039127923 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3268068a50 0x7f32680c9690 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.949+0000 7f326e4c3700 1 --2- 172.21.15.64:0/4039127923 >> v2:172.21.15.86:3300/0 conn(0x7f3268140c90 0x7f3268068510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.954 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.949+0000 7f326e4c3700 1 -- 172.21.15.64:0/4039127923 >> 172.21.15.64:0/4039127923 conn(0x7f326806a0b0 msgr2=0x7f326806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:13.955 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.949+0000 7f326e4c3700 1 -- 172.21.15.64:0/4039127923 shutdown_connections 2023-06-23T00:40:13.955 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.949+0000 7f326e4c3700 1 -- 172.21.15.64:0/4039127923 wait complete. 2023-06-23T00:40:13.957 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.950+0000 7f326e4c3700 1 Processor -- start 2023-06-23T00:40:13.957 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.950+0000 7f326e4c3700 1 -- start start 2023-06-23T00:40:13.957 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.950+0000 7f326e4c3700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3268068a50 0x7f32680a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.957 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f326e4c3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32681408c0 0x7f3268167050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.957 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f326e4c3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3268140c90 0x7f3268167590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f326e4c3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3268167cb0 con 0x7f32681408c0 2023-06-23T00:40:13.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f326e4c3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3268167ad0 con 0x7f3268140c90 2023-06-23T00:40:13.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f326e4c3700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32680b1fb0 con 0x7f3268068a50 2023-06-23T00:40:13.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f32677fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32681408c0 0x7f3268167050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f32677fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32681408c0 0x7f3268167050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51798/0 (socket says 172.21.15.64:51798) 2023-06-23T00:40:13.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f32677fe700 1 -- 172.21.15.64:0/2864124320 learned_addr learned my addr 172.21.15.64:0/2864124320 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:13.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f3267fff700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3268068a50 0x7f32680a4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f326ca60700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3268140c90 0x7f3268167590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.959 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f32677fe700 1 -- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3268068a50 msgr2=0x7f32680a4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.959 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f32677fe700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3268068a50 0x7f32680a4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.959 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f32677fe700 1 -- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3268140c90 msgr2=0x7f3268167590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:13.959 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f32677fe700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3268140c90 0x7f3268167590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:13.959 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f32677fe700 1 -- 172.21.15.64:0/2864124320 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3260009a20 con 0x7f32681408c0 2023-06-23T00:40:13.959 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f32677fe700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32681408c0 0x7f3268167050 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f325000ec50 tx=0x7f32500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:13.959 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f325ffff700 1 -- 172.21.15.64:0/2864124320 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325000c850 con 0x7f32681408c0 2023-06-23T00:40:13.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f325ffff700 1 -- 172.21.15.64:0/2864124320 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325000c9b0 con 0x7f32681408c0 2023-06-23T00:40:13.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f326e4c3700 1 -- 172.21.15.64:0/2864124320 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32680b2290 con 0x7f32681408c0 2023-06-23T00:40:13.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f325ffff700 1 -- 172.21.15.64:0/2864124320 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325000cd70 con 0x7f32681408c0 2023-06-23T00:40:13.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.951+0000 7f326e4c3700 1 -- 172.21.15.64:0/2864124320 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32680b2850 con 0x7f32681408c0 2023-06-23T00:40:13.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.952+0000 7f326e4c3700 1 -- 172.21.15.64:0/2864124320 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326804c1b0 con 0x7f32681408c0 2023-06-23T00:40:13.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.953+0000 7f325ffff700 1 -- 172.21.15.64:0/2864124320 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f3250013680 con 0x7f32681408c0 2023-06-23T00:40:13.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.953+0000 7f325ffff700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3248066e90 0x7f3248069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:13.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.953+0000 7f325ffff700 1 -- 172.21.15.64:0/2864124320 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3250084470 con 0x7f32681408c0 2023-06-23T00:40:13.961 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.953+0000 7f3267fff700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3248066e90 0x7f3248069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:13.961 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.954+0000 7f3267fff700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3248066e90 0x7f3248069340 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3258007fd0 tx=0x7f3258007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:13.961 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:13.957+0000 7f325ffff700 1 -- 172.21.15.64:0/2864124320 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f3250051990 con 0x7f32681408c0 2023-06-23T00:40:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:14.152+0000 7f326e4c3700 1 -- 172.21.15.64:0/2864124320 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f32680ad340 con 0x7f32681408c0 2023-06-23T00:40:15.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.000+0000 7f325ffff700 1 -- 172.21.15.64:0/2864124320 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3250054fb0 con 0x7f32681408c0 2023-06-23T00:40:15.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.005+0000 7f326e4c3700 1 -- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3248066e90 msgr2=0x7f3248069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:15.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.005+0000 7f326e4c3700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3248066e90 0x7f3248069340 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3258007fd0 tx=0x7f3258007d50 comp rx=0 tx=0).stop 2023-06-23T00:40:15.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.005+0000 7f326e4c3700 1 -- 172.21.15.64:0/2864124320 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32681408c0 msgr2=0x7f3268167050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:15.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.005+0000 7f326e4c3700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32681408c0 0x7f3268167050 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f325000ec50 tx=0x7f32500091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:15.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.007+0000 7f326e4c3700 1 -- 172.21.15.64:0/2864124320 shutdown_connections 2023-06-23T00:40:15.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.007+0000 7f326e4c3700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3248066e90 0x7f3248069340 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:15.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.007+0000 7f326e4c3700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3268140c90 0x7f3268167590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:15.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.007+0000 7f326e4c3700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3268068a50 0x7f32680a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:15.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.007+0000 7f326e4c3700 1 --2- 172.21.15.64:0/2864124320 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32681408c0 0x7f3268167050 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:15.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.007+0000 7f326e4c3700 1 -- 172.21.15.64:0/2864124320 >> 172.21.15.64:0/2864124320 conn(0x7f326806a0b0 msgr2=0x7f32680ad6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:15.011 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.008+0000 7f326e4c3700 1 -- 172.21.15.64:0/2864124320 shutdown_connections 2023-06-23T00:40:15.011 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.008+0000 7f326e4c3700 1 -- 172.21.15.64:0/2864124320 wait complete. 2023-06-23T00:40:15.012 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_metadata' created 2023-06-23T00:40:15.026 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-06-23T00:40:15.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.157+0000 7f9b12196700 1 Processor -- start 2023-06-23T00:40:15.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.157+0000 7f9b12196700 1 -- start start 2023-06-23T00:40:15.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b12196700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9b0c159590 0x7f9b0c159960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:15.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b12196700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f9b0c15e110 0x7f9b0c0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:15.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b12196700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0c0b07c0 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b12196700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0c0b25a0 con 0x7f9b0c15e110 2023-06-23T00:40:15.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b12196700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0c0b2880 con 0x7f9b0c159590 2023-06-23T00:40:15.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b0affd700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f9b0c15e110 0x7f9b0c0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:15.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b0affd700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f9b0c15e110 0x7f9b0c0aff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51810/0 (socket says 172.21.15.64:51810) 2023-06-23T00:40:15.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b0affd700 1 -- 172.21.15.64:0/2549165552 learned_addr learned my addr 172.21.15.64:0/2549165552 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:15.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b0b7fe700 1 --2- 172.21.15.64:0/2549165552 >> v2:172.21.15.86:3300/0 conn(0x7f9b0c159590 0x7f9b0c159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:15.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b0affd700 1 -- 172.21.15.64:0/2549165552 >> v2:172.21.15.86:3300/0 conn(0x7f9b0c159590 msgr2=0x7f9b0c159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:15.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b0affd700 1 --2- 172.21.15.64:0/2549165552 >> v2:172.21.15.86:3300/0 conn(0x7f9b0c159590 0x7f9b0c159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:15.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b0affd700 1 -- 172.21.15.64:0/2549165552 >> v1:172.21.15.86:6789/0 conn(0x7f9b0c0d02d0 legacy=0x7f9b0c0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:15.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b0affd700 1 -- 172.21.15.64:0/2549165552 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c0b0a60 con 0x7f9b0c15e110 2023-06-23T00:40:15.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.158+0000 7f9b0affd700 1 --2- 172.21.15.64:0/2549165552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b0c15e110 0x7f9b0c0aff20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0894d0 tx=0x7f9b00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f943d80dc542284 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:15.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.159+0000 7f9b0a7fc700 1 -- 172.21.15.64:0/2549165552 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b00009e70 con 0x7f9b0c15e110 2023-06-23T00:40:15.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.159+0000 7f9b0a7fc700 1 -- 172.21.15.64:0/2549165552 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0000b860 con 0x7f9b0c15e110 2023-06-23T00:40:15.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.159+0000 7f9b0a7fc700 1 -- 172.21.15.64:0/2549165552 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0000bc00 con 0x7f9b0c15e110 2023-06-23T00:40:15.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.159+0000 7f9b12196700 1 -- 172.21.15.64:0/2549165552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b0c15e110 msgr2=0x7f9b0c0aff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:15.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.159+0000 7f9b12196700 1 --2- 172.21.15.64:0/2549165552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b0c15e110 0x7f9b0c0aff20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0894d0 tx=0x7f9b00009250 comp rx=0 tx=0).stop 2023-06-23T00:40:15.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.159+0000 7f9b12196700 1 -- 172.21.15.64:0/2549165552 shutdown_connections 2023-06-23T00:40:15.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.159+0000 7f9b12196700 1 --2- 172.21.15.64:0/2549165552 >> v2:172.21.15.86:3300/0 conn(0x7f9b0c159590 0x7f9b0c159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:15.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.159+0000 7f9b12196700 1 --2- 172.21.15.64:0/2549165552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b0c15e110 0x7f9b0c0aff20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:15.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.159+0000 7f9b12196700 1 -- 172.21.15.64:0/2549165552 >> 172.21.15.64:0/2549165552 conn(0x7f9b0c06a170 msgr2=0x7f9b0c06a540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:15.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.159+0000 7f9b12196700 1 -- 172.21.15.64:0/2549165552 shutdown_connections 2023-06-23T00:40:15.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.159+0000 7f9b12196700 1 -- 172.21.15.64:0/2549165552 wait complete. 2023-06-23T00:40:15.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.160+0000 7f9b12196700 1 Processor -- start 2023-06-23T00:40:15.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.160+0000 7f9b12196700 1 -- start start 2023-06-23T00:40:15.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.161+0000 7f9b12196700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b0c0d02d0 0x7f9b0c16c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:15.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.161+0000 7f9b12196700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b0c159590 0x7f9b0c0a7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:15.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.161+0000 7f9b12196700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b0c15e110 0x7f9b0c15c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:15.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.161+0000 7f9b12196700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0ca4c0 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.161+0000 7f9b12196700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c16ce30 con 0x7f9b0c15e110 2023-06-23T00:40:15.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.161+0000 7f9b12196700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0c9840 con 0x7f9b0c159590 2023-06-23T00:40:15.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.161+0000 7f9b0b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b0c0d02d0 0x7f9b0c16c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:15.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.161+0000 7f9b0b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b0c0d02d0 0x7f9b0c16c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51816/0 (socket says 172.21.15.64:51816) 2023-06-23T00:40:15.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.161+0000 7f9b0b7fe700 1 -- 172.21.15.64:0/2009574792 learned_addr learned my addr 172.21.15.64:0/2009574792 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:15.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.161+0000 7f9b0bfff700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b0c15e110 0x7f9b0c15c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:15.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.161+0000 7f9b0affd700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b0c159590 0x7f9b0c0a7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:15.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.162+0000 7f9b0b7fe700 1 -- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b0c159590 msgr2=0x7f9b0c0a7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:15.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.162+0000 7f9b0b7fe700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b0c159590 0x7f9b0c0a7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:15.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.162+0000 7f9b0b7fe700 1 -- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b0c15e110 msgr2=0x7f9b0c15c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:15.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.162+0000 7f9b0b7fe700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b0c15e110 0x7f9b0c15c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:15.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.162+0000 7f9b0b7fe700 1 -- 172.21.15.64:0/2009574792 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b000094f0 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.162+0000 7f9b0b7fe700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b0c0d02d0 0x7f9b0c16c8f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00ec80 tx=0x7f9afc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:15.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.162+0000 7f9af37fe700 1 -- 172.21.15.64:0/2009574792 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc00c850 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.162+0000 7f9af37fe700 1 -- 172.21.15.64:0/2009574792 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc00c9b0 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.162+0000 7f9b12196700 1 -- 172.21.15.64:0/2009574792 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c0c9b20 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.162+0000 7f9af37fe700 1 -- 172.21.15.64:0/2009574792 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc00cd50 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.162+0000 7f9b12196700 1 -- 172.21.15.64:0/2009574792 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0c0ca0e0 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.163+0000 7f9b12196700 1 -- 172.21.15.64:0/2009574792 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0c071370 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.163+0000 7f9af37fe700 1 -- 172.21.15.64:0/2009574792 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f9afc013680 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.164+0000 7f9af37fe700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9af8066e90 0x7f9af8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:15.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.164+0000 7f9af37fe700 1 -- 172.21.15.64:0/2009574792 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5576+0+0 (secure 0 0 0) 0x7f9afc0845f0 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.164+0000 7f9b0affd700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9af8066e90 0x7f9af8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:15.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.164+0000 7f9b0affd700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9af8066e90 0x7f9af8069340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0894d0 tx=0x7f9b00017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:15.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.168+0000 7f9af37fe700 1 -- 172.21.15.64:0/2009574792 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f9afc0519a0 con 0x7f9b0c0d02d0 2023-06-23T00:40:15.368 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:15.365+0000 7f9b12196700 1 -- 172.21.15.64:0/2009574792 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f9b0c0cacb0 con 0x7f9b0c0d02d0 2023-06-23T00:40:16.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.011+0000 7f9af37fe700 1 -- 172.21.15.64:0/2009574792 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f9afc054fc0 con 0x7f9b0c0d02d0 2023-06-23T00:40:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.017+0000 7f9b12196700 1 -- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9af8066e90 msgr2=0x7f9af8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.017+0000 7f9b12196700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9af8066e90 0x7f9af8069340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0894d0 tx=0x7f9b00017000 comp rx=0 tx=0).stop 2023-06-23T00:40:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.017+0000 7f9b12196700 1 -- 172.21.15.64:0/2009574792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b0c0d02d0 msgr2=0x7f9b0c16c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.017+0000 7f9b12196700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b0c0d02d0 0x7f9b0c16c8f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9afc00ec80 tx=0x7f9afc0091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.018+0000 7f9b12196700 1 -- 172.21.15.64:0/2009574792 shutdown_connections 2023-06-23T00:40:16.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.018+0000 7f9b12196700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f9af8066e90 0x7f9af8069340 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.018+0000 7f9b12196700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b0c15e110 0x7f9b0c15c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.018+0000 7f9b12196700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9b0c0d02d0 0x7f9b0c16c8f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.018+0000 7f9b12196700 1 --2- 172.21.15.64:0/2009574792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b0c159590 0x7f9b0c0a7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.018+0000 7f9b12196700 1 -- 172.21.15.64:0/2009574792 >> 172.21.15.64:0/2009574792 conn(0x7f9b0c06a170 msgr2=0x7f9b0c0cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:16.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.019+0000 7f9b12196700 1 -- 172.21.15.64:0/2009574792 shutdown_connections 2023-06-23T00:40:16.023 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.019+0000 7f9b12196700 1 -- 172.21.15.64:0/2009574792 wait complete. 2023-06-23T00:40:16.024 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data' created 2023-06-23T00:40:16.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-23T00:40:16.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.164+0000 7f4138f7c700 1 Processor -- start 2023-06-23T00:40:16.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.164+0000 7f4138f7c700 1 -- start start 2023-06-23T00:40:16.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.165+0000 7f4138f7c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f41340d04f0 0x7f413415a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.165+0000 7f4138f7c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 0x7f41340cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.165+0000 7f4138f7c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413415aaa0 con 0x7f41341569b0 2023-06-23T00:40:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.165+0000 7f4138f7c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41340ce5c0 con 0x7f41340d04f0 2023-06-23T00:40:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.165+0000 7f4138f7c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41340ccf70 con 0x7f41340d08c0 2023-06-23T00:40:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f413259c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f41340d04f0 0x7f413415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f4131d9b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 0x7f41340cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f4132d9d700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f41341569b0 0x7f413415cf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:39298/0 (socket says 172.21.15.64:39298) 2023-06-23T00:40:16.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f4132d9d700 1 -- 172.21.15.64:0/1288026902 learned_addr learned my addr 172.21.15.64:0/1288026902 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:16.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f413159a700 1 -- 172.21.15.64:0/1288026902 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729350536 0 0) 0x7f413415aaa0 con 0x7f41341569b0 2023-06-23T00:40:16.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f413159a700 1 -- 172.21.15.64:0/1288026902 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4120003570 con 0x7f41341569b0 2023-06-23T00:40:16.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f4131d9b700 1 -- 172.21.15.64:0/1288026902 >> v2:172.21.15.86:3300/0 conn(0x7f41340d04f0 msgr2=0x7f413415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:16.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f4131d9b700 1 --2- 172.21.15.64:0/1288026902 >> v2:172.21.15.86:3300/0 conn(0x7f41340d04f0 0x7f413415a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f4131d9b700 1 -- 172.21.15.64:0/1288026902 >> v1:172.21.15.86:6789/0 conn(0x7f41341569b0 legacy=0x7f413415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f4131d9b700 1 -- 172.21.15.64:0/1288026902 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41340cd1d0 con 0x7f41340d08c0 2023-06-23T00:40:16.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f413259c700 1 --2- 172.21.15.64:0/1288026902 >> v2:172.21.15.86:3300/0 conn(0x7f41340d04f0 0x7f413415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:16.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f4131d9b700 1 --2- 172.21.15.64:0/1288026902 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 0x7f41340cca30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f411c0098e0 tx=0x7f411c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=534eb5fcdbe26e2f server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:16.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f413159a700 1 -- 172.21.15.64:0/1288026902 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c011630 con 0x7f41340d08c0 2023-06-23T00:40:16.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f413159a700 1 -- 172.21.15.64:0/1288026902 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c011790 con 0x7f41340d08c0 2023-06-23T00:40:16.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.166+0000 7f413159a700 1 -- 172.21.15.64:0/1288026902 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c011b10 con 0x7f41340d08c0 2023-06-23T00:40:16.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.167+0000 7f4138f7c700 1 -- 172.21.15.64:0/1288026902 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 msgr2=0x7f41340cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.167+0000 7f4138f7c700 1 --2- 172.21.15.64:0/1288026902 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 0x7f41340cca30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f411c0098e0 tx=0x7f411c0091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.167+0000 7f4138f7c700 1 -- 172.21.15.64:0/1288026902 shutdown_connections 2023-06-23T00:40:16.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.167+0000 7f4138f7c700 1 --2- 172.21.15.64:0/1288026902 >> v2:172.21.15.86:3300/0 conn(0x7f41340d04f0 0x7f413415a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.167+0000 7f4138f7c700 1 --2- 172.21.15.64:0/1288026902 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 0x7f41340cca30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.167+0000 7f4138f7c700 1 -- 172.21.15.64:0/1288026902 >> 172.21.15.64:0/1288026902 conn(0x7f413406a060 msgr2=0x7f413406a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:16.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.167+0000 7f4138f7c700 1 -- 172.21.15.64:0/1288026902 shutdown_connections 2023-06-23T00:40:16.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.167+0000 7f4138f7c700 1 -- 172.21.15.64:0/1288026902 wait complete. 2023-06-23T00:40:16.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.168+0000 7f4138f7c700 1 Processor -- start 2023-06-23T00:40:16.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.168+0000 7f4138f7c700 1 -- start start 2023-06-23T00:40:16.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4138f7c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41340d04f0 0x7f4134154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4138f7c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 0x7f413415f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4138f7c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41341569b0 0x7f413415f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4138f7c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41340ca040 con 0x7f41341569b0 2023-06-23T00:40:16.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4138f7c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41340b06c0 con 0x7f41340d04f0 2023-06-23T00:40:16.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4138f7c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4134155460 con 0x7f41340d08c0 2023-06-23T00:40:16.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4132d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41341569b0 0x7f413415f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4132d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41341569b0 0x7f413415f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51822/0 (socket says 172.21.15.64:51822) 2023-06-23T00:40:16.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4132d9d700 1 -- 172.21.15.64:0/1654356646 learned_addr learned my addr 172.21.15.64:0/1654356646 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:16.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f413259c700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41340d04f0 0x7f4134154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4131d9b700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 0x7f413415f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4132d9d700 1 -- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 msgr2=0x7f413415f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4132d9d700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 0x7f413415f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4132d9d700 1 -- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41340d04f0 msgr2=0x7f4134154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4132d9d700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41340d04f0 0x7f4134154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4132d9d700 1 -- 172.21.15.64:0/1654356646 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4124009a20 con 0x7f41341569b0 2023-06-23T00:40:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f413259c700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41340d04f0 0x7f4134154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T00:40:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4131d9b700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 0x7f413415f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T00:40:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.169+0000 7f4132d9d700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41341569b0 0x7f413415f780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f412c00c560 tx=0x7f412c007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.170+0000 7f412a7fc700 1 -- 172.21.15.64:0/1654356646 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c005c00 con 0x7f41341569b0 2023-06-23T00:40:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.170+0000 7f412a7fc700 1 -- 172.21.15.64:0/1654356646 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412c005d60 con 0x7f41341569b0 2023-06-23T00:40:16.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.170+0000 7f4138f7c700 1 -- 172.21.15.64:0/1654356646 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411c009420 con 0x7f41341569b0 2023-06-23T00:40:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.170+0000 7f412a7fc700 1 -- 172.21.15.64:0/1654356646 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412c0072e0 con 0x7f41341569b0 2023-06-23T00:40:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.170+0000 7f4138f7c700 1 -- 172.21.15.64:0/1654356646 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41340b0070 con 0x7f41341569b0 2023-06-23T00:40:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.171+0000 7f4138f7c700 1 -- 172.21.15.64:0/1654356646 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413404c1b0 con 0x7f41341569b0 2023-06-23T00:40:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.171+0000 7f412a7fc700 1 -- 172.21.15.64:0/1654356646 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f412c012400 con 0x7f41341569b0 2023-06-23T00:40:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.171+0000 7f412a7fc700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4114066e90 0x7f4114069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.171+0000 7f412a7fc700 1 -- 172.21.15.64:0/1654356646 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5958+0+0 (secure 0 0 0) 0x7f412c083970 con 0x7f41341569b0 2023-06-23T00:40:16.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.172+0000 7f413259c700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4114066e90 0x7f4114069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.172+0000 7f413259c700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4114066e90 0x7f4114069340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f41240099f0 tx=0x7f412400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:16.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.176+0000 7f412a7fc700 1 -- 172.21.15.64:0/1654356646 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f412c04b590 con 0x7f41341569b0 2023-06-23T00:40:16.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.413+0000 7f4138f7c700 1 -- 172.21.15.64:0/1654356646 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f41340cd4b0 con 0x7f41341569b0 2023-06-23T00:40:16.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.419+0000 7f412a7fc700 1 -- 172.21.15.64:0/1654356646 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f412c0170c0 con 0x7f41341569b0 2023-06-23T00:40:16.426 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.424+0000 7f4138f7c700 1 -- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4114066e90 msgr2=0x7f4114069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.427 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.424+0000 7f4138f7c700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4114066e90 0x7f4114069340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f41240099f0 tx=0x7f412400b040 comp rx=0 tx=0).stop 2023-06-23T00:40:16.427 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.424+0000 7f4138f7c700 1 -- 172.21.15.64:0/1654356646 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41341569b0 msgr2=0x7f413415f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.427 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.424+0000 7f4138f7c700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41341569b0 0x7f413415f780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f412c00c560 tx=0x7f412c007dd0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.427 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.424+0000 7f4138f7c700 1 -- 172.21.15.64:0/1654356646 shutdown_connections 2023-06-23T00:40:16.427 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.424+0000 7f4138f7c700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f4114066e90 0x7f4114069340 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.427 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.424+0000 7f4138f7c700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41341569b0 0x7f413415f780 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.427 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.424+0000 7f4138f7c700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41340d04f0 0x7f4134154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.428 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.424+0000 7f4138f7c700 1 --2- 172.21.15.64:0/1654356646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41340d08c0 0x7f413415f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.428 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.424+0000 7f4138f7c700 1 -- 172.21.15.64:0/1654356646 >> 172.21.15.64:0/1654356646 conn(0x7f413406a060 msgr2=0x7f4134155bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:16.428 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.425+0000 7f4138f7c700 1 -- 172.21.15.64:0/1654356646 shutdown_connections 2023-06-23T00:40:16.431 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.426+0000 7f4138f7c700 1 -- 172.21.15.64:0/1654356646 wait complete. 2023-06-23T00:40:16.444 INFO:teuthology.orchestra.run.smithi064.stderr:new fs with metadata pool 3 and data pool 4 2023-06-23T00:40:16.453 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-06-23T00:40:16.454 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-06-23T00:40:16.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.576+0000 7f41e5ac5700 1 Processor -- start 2023-06-23T00:40:16.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.576+0000 7f41e5ac5700 1 -- start start 2023-06-23T00:40:16.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.576+0000 7f41e5ac5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e006dfc0 0x7f41e00b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.576+0000 7f41e5ac5700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e015abc0 con 0x7f41e015a7f0 2023-06-23T00:40:16.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.576+0000 7f41e5ac5700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e00b0e20 con 0x7f41e00b22c0 2023-06-23T00:40:16.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.576+0000 7f41e5ac5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e00cd270 con 0x7f41e006dfc0 2023-06-23T00:40:16.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.576+0000 7f41df7fe700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f41e015a7f0 0x7f41e006d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36682/0 (socket says 172.21.15.64:36682) 2023-06-23T00:40:16.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41df7fe700 1 -- 172.21.15.64:0/2509399593 learned_addr learned my addr 172.21.15.64:0/2509399593 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41deffd700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e006dfc0 0x7f41e00b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144538255 0 0) 0x7f41e015abc0 con 0x7f41e015a7f0 2023-06-23T00:40:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c8003570 con 0x7f41e015a7f0 2023-06-23T00:40:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2656620066 0 0) 0x7f41c8003570 con 0x7f41e015a7f0 2023-06-23T00:40:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e006dfc0 msgr2=0x7f41e00b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e006dfc0 0x7f41e00b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 >> v1:172.21.15.86:6789/0 conn(0x7f41e00b22c0 legacy=0x7f41e0160e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e00cd550 con 0x7f41e015a7f0 2023-06-23T00:40:16.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 939283155 0 0) 0x7f41d0001470 con 0x7f41e015a7f0 2023-06-23T00:40:16.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 >> v1:172.21.15.64:6789/0 conn(0x7f41e015a7f0 legacy=0x7f41e006d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e006dfc0 0x7f41c8003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41c8003fd0 0x7f41c8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8004d50 0x7f41c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e00cd270 con 0x7f41c8004d50 2023-06-23T00:40:16.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c800bb40 con 0x7f41c8003fd0 2023-06-23T00:40:16.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c800bd00 con 0x7f41e006dfc0 2023-06-23T00:40:16.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41dffff700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8004d50 0x7f41c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41deffd700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41c8003fd0 0x7f41c8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.577+0000 7f41df7fe700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e006dfc0 0x7f41c8003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41deffd700 1 -- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e006dfc0 msgr2=0x7f41c8003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:16.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41deffd700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e006dfc0 0x7f41c8003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41deffd700 1 -- 172.21.15.64:0/2509399593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8004d50 msgr2=0x7f41c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:16.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41deffd700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8004d50 0x7f41c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41deffd700 1 -- 172.21.15.64:0/2509399593 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c8003570 con 0x7f41c8003fd0 2023-06-23T00:40:16.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41dffff700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8004d50 0x7f41c80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:16.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41deffd700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41c8003fd0 0x7f41c8004810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f41e00cf130 tx=0x7f41d40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14c677ad109c75f8 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:16.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d400e710 con 0x7f41c8003fd0 2023-06-23T00:40:16.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d400e870 con 0x7f41c8003fd0 2023-06-23T00:40:16.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41de7fc700 1 -- 172.21.15.64:0/2509399593 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d400ec30 con 0x7f41c8003fd0 2023-06-23T00:40:16.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41e5ac5700 1 -- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41c8003fd0 msgr2=0x7f41c8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41e5ac5700 1 --2- 172.21.15.64:0/2509399593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41c8003fd0 0x7f41c8004810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f41e00cf130 tx=0x7f41d40091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41dffff700 1 -- 172.21.15.64:0/2509399593 reap_dead start 2023-06-23T00:40:16.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41e5ac5700 1 -- 172.21.15.64:0/2509399593 shutdown_connections 2023-06-23T00:40:16.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41e5ac5700 1 -- 172.21.15.64:0/2509399593 >> 172.21.15.64:0/2509399593 conn(0x7f41e006cba0 msgr2=0x7f41e00afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:16.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41e5ac5700 1 -- 172.21.15.64:0/2509399593 shutdown_connections 2023-06-23T00:40:16.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.578+0000 7f41e5ac5700 1 -- 172.21.15.64:0/2509399593 wait complete. 2023-06-23T00:40:16.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.580+0000 7f41e5ac5700 1 Processor -- start 2023-06-23T00:40:16.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.580+0000 7f41e5ac5700 1 -- start start 2023-06-23T00:40:16.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.580+0000 7f41e5ac5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8003fd0 0x7f41e01567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.580+0000 7f41e5ac5700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e01672d0 0x7f41e015abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.580+0000 7f41e5ac5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41e0167740 0x7f41e015b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.580+0000 7f41e5ac5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e00c4ea0 con 0x7f41c8003fd0 2023-06-23T00:40:16.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.580+0000 7f41e5ac5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e00c5420 con 0x7f41e0167740 2023-06-23T00:40:16.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.580+0000 7f41e5ac5700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e00c7460 con 0x7f41e01672d0 2023-06-23T00:40:16.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41df7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8003fd0 0x7f41e01567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41df7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8003fd0 0x7f41e01567b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51850/0 (socket says 172.21.15.64:51850) 2023-06-23T00:40:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41deffd700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e01672d0 0x7f41e015abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41df7fe700 1 -- 172.21.15.64:0/3545002477 learned_addr learned my addr 172.21.15.64:0/3545002477 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41dffff700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41e0167740 0x7f41e015b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41df7fe700 1 -- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e01672d0 msgr2=0x7f41e015abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41df7fe700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e01672d0 0x7f41e015abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41df7fe700 1 -- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41e0167740 msgr2=0x7f41e015b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:16.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41df7fe700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41e0167740 0x7f41e015b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:16.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41df7fe700 1 -- 172.21.15.64:0/3545002477 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e00c7740 con 0x7f41c8003fd0 2023-06-23T00:40:16.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41df7fe700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8003fd0 0x7f41e01567b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f41d0007390 tx=0x7f41d0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:16.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41c77fe700 1 -- 172.21.15.64:0/3545002477 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d0018030 con 0x7f41c8003fd0 2023-06-23T00:40:16.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41c77fe700 1 -- 172.21.15.64:0/3545002477 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d0007750 con 0x7f41c8003fd0 2023-06-23T00:40:16.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41e5ac5700 1 -- 172.21.15.64:0/3545002477 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e00c50b0 con 0x7f41c8003fd0 2023-06-23T00:40:16.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41c77fe700 1 -- 172.21.15.64:0/3545002477 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d000faa0 con 0x7f41c8003fd0 2023-06-23T00:40:16.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.581+0000 7f41e5ac5700 1 -- 172.21.15.64:0/3545002477 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e00c6750 con 0x7f41c8003fd0 2023-06-23T00:40:16.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.582+0000 7f41e5ac5700 1 -- 172.21.15.64:0/3545002477 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e00c7930 con 0x7f41c8003fd0 2023-06-23T00:40:16.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.583+0000 7f41c77fe700 1 -- 172.21.15.64:0/3545002477 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f41d001b070 con 0x7f41c8003fd0 2023-06-23T00:40:16.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.583+0000 7f41c77fe700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f41cc066e90 0x7f41cc069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:16.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.583+0000 7f41c77fe700 1 -- 172.21.15.64:0/3545002477 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f41d0019070 con 0x7f41c8003fd0 2023-06-23T00:40:16.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.583+0000 7f41deffd700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f41cc066e90 0x7f41cc069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:16.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.584+0000 7f41deffd700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f41cc066e90 0x7f41cc069340 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41d40091e0 tx=0x7f41d401a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:16.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.587+0000 7f41c77fe700 1 -- 172.21.15.64:0/3545002477 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f41e00c7930 con 0x7f41c8003fd0 2023-06-23T00:40:16.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:16.792+0000 7f41e5ac5700 1 -- 172.21.15.64:0/3545002477 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f41e00c78d0 con 0x7f41c8003fd0 2023-06-23T00:40:17.425 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.423+0000 7f41c77fe700 1 -- 172.21.15.64:0/3545002477 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7f41d00515f0 con 0x7f41c8003fd0 2023-06-23T00:40:17.428 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 -- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f41cc066e90 msgr2=0x7f41cc069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f41cc066e90 0x7f41cc069340 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41d40091e0 tx=0x7f41d401a000 comp rx=0 tx=0).stop 2023-06-23T00:40:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 -- 172.21.15.64:0/3545002477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8003fd0 msgr2=0x7f41e01567b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8003fd0 0x7f41e01567b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f41d0007390 tx=0x7f41d0001e70 comp rx=0 tx=0).stop 2023-06-23T00:40:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 -- 172.21.15.64:0/3545002477 shutdown_connections 2023-06-23T00:40:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f41cc066e90 0x7f41cc069340 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:17.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41e0167740 0x7f41e015b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:17.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41c8003fd0 0x7f41e01567b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:17.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 --2- 172.21.15.64:0/3545002477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41e01672d0 0x7f41e015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:17.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 -- 172.21.15.64:0/3545002477 >> 172.21.15.64:0/3545002477 conn(0x7f41e006cba0 msgr2=0x7f41e00aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:17.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 -- 172.21.15.64:0/3545002477 shutdown_connections 2023-06-23T00:40:17.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.426+0000 7f41e5ac5700 1 -- 172.21.15.64:0/3545002477 wait complete. 2023-06-23T00:40:17.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-06-23T00:40:17.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.568+0000 7faf21a2f700 1 Processor -- start 2023-06-23T00:40:17.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.568+0000 7faf21a2f700 1 -- start start 2023-06-23T00:40:17.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf21a2f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf1c06a3c0 0x7faf1c06a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:17.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf21a2f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faf1c0cfb50 0x7faf1c0cb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:17.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf21a2f700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf1c0cf6c0 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf21a2f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf1c0b1050 con 0x7faf1c0cfb50 2023-06-23T00:40:17.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf21a2f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c0d0d40 con 0x7faf1c06a3c0 2023-06-23T00:40:17.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf1b7fe700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7faf1c0cb8c0 0x7faf1c158fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:36690/0 (socket says 172.21.15.64:36690) 2023-06-23T00:40:17.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf1b7fe700 1 -- 172.21.15.64:0/3990990402 learned_addr learned my addr 172.21.15.64:0/3990990402 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:17.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf1affd700 1 --2- 172.21.15.64:0/3990990402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf1c06a3c0 0x7faf1c06a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:17.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf1a7fc700 1 --2- 172.21.15.64:0/3990990402 >> v2:172.21.15.86:3300/0 conn(0x7faf1c0cfb50 0x7faf1c0cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:17.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf19ffb700 1 -- 172.21.15.64:0/3990990402 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677109608 0 0) 0x7faf1c0cf6c0 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf19ffb700 1 -- 172.21.15.64:0/3990990402 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf04003570 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf1a7fc700 1 -- 172.21.15.64:0/3990990402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf1c06a3c0 msgr2=0x7faf1c06a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:17.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf1a7fc700 1 --2- 172.21.15.64:0/3990990402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf1c06a3c0 0x7faf1c06a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:17.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf19ffb700 1 -- 172.21.15.64:0/3990990402 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3875857838 0 0) 0x7faf04003570 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf1a7fc700 1 -- 172.21.15.64:0/3990990402 >> v1:172.21.15.64:6789/0 conn(0x7faf1c0cb8c0 legacy=0x7faf1c158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:17.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.569+0000 7faf1a7fc700 1 -- 172.21.15.64:0/3990990402 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1c0ceab0 con 0x7faf1c0cfb50 2023-06-23T00:40:17.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf1affd700 1 --2- 172.21.15.64:0/3990990402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf1c06a3c0 0x7faf1c06a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:17.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf1a7fc700 1 --2- 172.21.15.64:0/3990990402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1c0cfb50 0x7faf1c0cb380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf100098e0 tx=0x7faf100091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71af18e8f65e2b35 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:17.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf19ffb700 1 -- 172.21.15.64:0/3990990402 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf140017f0 con 0x7faf1c0cfb50 2023-06-23T00:40:17.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf19ffb700 1 -- 172.21.15.64:0/3990990402 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf10011420 con 0x7faf1c0cfb50 2023-06-23T00:40:17.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf19ffb700 1 -- 172.21.15.64:0/3990990402 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf100117c0 con 0x7faf1c0cfb50 2023-06-23T00:40:17.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf21a2f700 1 -- 172.21.15.64:0/3990990402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1c0cfb50 msgr2=0x7faf1c0cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:17.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf21a2f700 1 --2- 172.21.15.64:0/3990990402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1c0cfb50 0x7faf1c0cb380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf100098e0 tx=0x7faf100091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:17.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf21a2f700 1 -- 172.21.15.64:0/3990990402 shutdown_connections 2023-06-23T00:40:17.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf21a2f700 1 --2- 172.21.15.64:0/3990990402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf1c06a3c0 0x7faf1c06a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:17.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf21a2f700 1 --2- 172.21.15.64:0/3990990402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1c0cfb50 0x7faf1c0cb380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:17.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf21a2f700 1 -- 172.21.15.64:0/3990990402 >> 172.21.15.64:0/3990990402 conn(0x7faf1c068f80 msgr2=0x7faf1c0d03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:17.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf21a2f700 1 -- 172.21.15.64:0/3990990402 shutdown_connections 2023-06-23T00:40:17.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.570+0000 7faf21a2f700 1 -- 172.21.15.64:0/3990990402 wait complete. 2023-06-23T00:40:17.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.572+0000 7faf21a2f700 1 Processor -- start 2023-06-23T00:40:17.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.572+0000 7faf21a2f700 1 -- start start 2023-06-23T00:40:17.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.572+0000 7faf21a2f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1c06a3c0 0x7faf1c149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:17.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.572+0000 7faf21a2f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf1c0cb8c0 0x7faf1c155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:17.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.572+0000 7faf21a2f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf1c0cfb50 0x7faf1c156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:17.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.572+0000 7faf21a2f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c149540 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.572+0000 7faf21a2f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c0b52f0 con 0x7faf1c06a3c0 2023-06-23T00:40:17.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.572+0000 7faf21a2f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c0b4400 con 0x7faf1c0cfb50 2023-06-23T00:40:17.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1a7fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf1c0cb8c0 0x7faf1c155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:17.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1affd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1c06a3c0 0x7faf1c149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:17.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1a7fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf1c0cb8c0 0x7faf1c155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:51864/0 (socket says 172.21.15.64:51864) 2023-06-23T00:40:17.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1a7fc700 1 -- 172.21.15.64:0/2772955293 learned_addr learned my addr 172.21.15.64:0/2772955293 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:17.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1b7fe700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf1c0cfb50 0x7faf1c156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:17.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1a7fc700 1 -- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf1c0cfb50 msgr2=0x7faf1c156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:17.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1a7fc700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf1c0cfb50 0x7faf1c156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:17.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1a7fc700 1 -- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1c06a3c0 msgr2=0x7faf1c149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:17.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1a7fc700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1c06a3c0 0x7faf1c149000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:17.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1a7fc700 1 -- 172.21.15.64:0/2772955293 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c009420 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1affd700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1c06a3c0 0x7faf1c149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:17.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf1a7fc700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf1c0cb8c0 0x7faf1c155bc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faf10002000 tx=0x7faf10016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:17.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf02ffd700 1 -- 172.21.15.64:0/2772955293 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1001caf0 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf02ffd700 1 -- 172.21.15.64:0/2772955293 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1001cc90 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf21a2f700 1 -- 172.21.15.64:0/2772955293 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf10009420 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf02ffd700 1 -- 172.21.15.64:0/2772955293 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1001a6a0 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.573+0000 7faf21a2f700 1 -- 172.21.15.64:0/2772955293 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1c0b4a40 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.574+0000 7faf21a2f700 1 -- 172.21.15.64:0/2772955293 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1c071370 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.575+0000 7faf02ffd700 1 -- 172.21.15.64:0/2772955293 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7faf1001aa50 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.575+0000 7faf02ffd700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7faf08066e90 0x7faf08069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:17.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.575+0000 7faf02ffd700 1 -- 172.21.15.64:0/2772955293 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6284+0+0 (secure 0 0 0) 0x7faf100859b0 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.575+0000 7faf1affd700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7faf08066e90 0x7faf08069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:17.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.576+0000 7faf1affd700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7faf08066e90 0x7faf08069340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faf0c011600 tx=0x7faf0c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:17.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.579+0000 7faf02ffd700 1 -- 172.21.15.64:0/2772955293 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7faf10052b50 con 0x7faf1c0cb8c0 2023-06-23T00:40:17.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:17.776+0000 7faf21a2f700 1 -- 172.21.15.64:0/2772955293 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7faf1c0ced90 con 0x7faf1c0cb8c0 2023-06-23T00:40:19.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.432+0000 7faf02ffd700 1 -- 172.21.15.64:0/2772955293 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7faf10029100 con 0x7faf1c0cb8c0 2023-06-23T00:40:19.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.437+0000 7faf21a2f700 1 -- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7faf08066e90 msgr2=0x7faf08069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.437+0000 7faf21a2f700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7faf08066e90 0x7faf08069340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faf0c011600 tx=0x7faf0c0091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.437+0000 7faf21a2f700 1 -- 172.21.15.64:0/2772955293 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf1c0cb8c0 msgr2=0x7faf1c155bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:19.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.437+0000 7faf21a2f700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf1c0cb8c0 0x7faf1c155bc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faf10002000 tx=0x7faf10016000 comp rx=0 tx=0).stop 2023-06-23T00:40:19.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.443+0000 7faf21a2f700 1 -- 172.21.15.64:0/2772955293 shutdown_connections 2023-06-23T00:40:19.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.443+0000 7faf21a2f700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7faf08066e90 0x7faf08069340 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.443+0000 7faf21a2f700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf1c0cfb50 0x7faf1c156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.443+0000 7faf21a2f700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf1c06a3c0 0x7faf1c149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.443+0000 7faf21a2f700 1 --2- 172.21.15.64:0/2772955293 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf1c0cb8c0 0x7faf1c155bc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.443+0000 7faf21a2f700 1 -- 172.21.15.64:0/2772955293 >> 172.21.15.64:0/2772955293 conn(0x7faf1c068f80 msgr2=0x7faf1c156b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:19.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.444+0000 7faf21a2f700 1 -- 172.21.15.64:0/2772955293 shutdown_connections 2023-06-23T00:40:19.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.444+0000 7faf21a2f700 1 -- 172.21.15.64:0/2772955293 wait complete. 2023-06-23T00:40:19.448 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data_ec' created 2023-06-23T00:40:19.462 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-06-23T00:40:19.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.587+0000 7fc771c20700 1 Processor -- start 2023-06-23T00:40:19.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.587+0000 7fc771c20700 1 -- start start 2023-06-23T00:40:19.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc771c20700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc76c0ccd40 0x7fc76c0cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:19.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc771c20700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fc76c0cf970 0x7fc76c0cde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:19.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc771c20700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc76c0ce390 0x7fc76c0aee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:19.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc771c20700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc76c0cf4e0 con 0x7fc76c0cf970 2023-06-23T00:40:19.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc771c20700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc76c0ce8d0 con 0x7fc76c0ccd40 2023-06-23T00:40:19.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc771c20700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c0cebb0 con 0x7fc76c0ce390 2023-06-23T00:40:19.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76affd700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fc76c0cf970 0x7fc76c0cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:19.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76affd700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7fc76c0cf970 0x7fc76c0cde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:40988/0 (socket says 172.21.15.64:40988) 2023-06-23T00:40:19.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76affd700 1 -- 172.21.15.64:0/2564215181 learned_addr learned my addr 172.21.15.64:0/2564215181 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:19.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76b7fe700 1 --2- 172.21.15.64:0/2564215181 >> v2:172.21.15.86:3300/0 conn(0x7fc76c0ccd40 0x7fc76c0cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:19.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76bfff700 1 --2- 172.21.15.64:0/2564215181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc76c0ce390 0x7fc76c0aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:19.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76affd700 1 -- 172.21.15.64:0/2564215181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc76c0ce390 msgr2=0x7fc76c0aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:19.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76affd700 1 --2- 172.21.15.64:0/2564215181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc76c0ce390 0x7fc76c0aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76affd700 1 -- 172.21.15.64:0/2564215181 >> v2:172.21.15.86:3300/0 conn(0x7fc76c0ccd40 msgr2=0x7fc76c0cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:19.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76affd700 1 --2- 172.21.15.64:0/2564215181 >> v2:172.21.15.86:3300/0 conn(0x7fc76c0ccd40 0x7fc76c0cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76affd700 1 -- 172.21.15.64:0/2564215181 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c0cee90 con 0x7fc76c0cf970 2023-06-23T00:40:19.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76affd700 1 --2- 172.21.15.64:0/2564215181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc76c0cf970 0x7fc76c0cde50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc76c089480 tx=0x7fc760009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86f7cdd7d22ce9d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:19.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.588+0000 7fc76a7fc700 1 -- 172.21.15.64:0/2564215181 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76000b860 con 0x7fc76c0cf970 2023-06-23T00:40:19.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.589+0000 7fc76a7fc700 1 -- 172.21.15.64:0/2564215181 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76000b9c0 con 0x7fc76c0cf970 2023-06-23T00:40:19.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.589+0000 7fc76a7fc700 1 -- 172.21.15.64:0/2564215181 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76000bd60 con 0x7fc76c0cf970 2023-06-23T00:40:19.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.589+0000 7fc771c20700 1 -- 172.21.15.64:0/2564215181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc76c0cf970 msgr2=0x7fc76c0cde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:19.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.589+0000 7fc771c20700 1 --2- 172.21.15.64:0/2564215181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc76c0cf970 0x7fc76c0cde50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc76c089480 tx=0x7fc760009250 comp rx=0 tx=0).stop 2023-06-23T00:40:19.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.589+0000 7fc771c20700 1 -- 172.21.15.64:0/2564215181 shutdown_connections 2023-06-23T00:40:19.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.589+0000 7fc771c20700 1 --2- 172.21.15.64:0/2564215181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc76c0ce390 0x7fc76c0aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.589+0000 7fc771c20700 1 --2- 172.21.15.64:0/2564215181 >> v2:172.21.15.86:3300/0 conn(0x7fc76c0ccd40 0x7fc76c0cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.589+0000 7fc771c20700 1 --2- 172.21.15.64:0/2564215181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc76c0cf970 0x7fc76c0cde50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.589+0000 7fc771c20700 1 -- 172.21.15.64:0/2564215181 >> 172.21.15.64:0/2564215181 conn(0x7fc76c087c70 msgr2=0x7fc76c068510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:19.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.589+0000 7fc771c20700 1 -- 172.21.15.64:0/2564215181 shutdown_connections 2023-06-23T00:40:19.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.589+0000 7fc771c20700 1 -- 172.21.15.64:0/2564215181 wait complete. 2023-06-23T00:40:19.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.590+0000 7fc771c20700 1 Processor -- start 2023-06-23T00:40:19.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.590+0000 7fc771c20700 1 -- start start 2023-06-23T00:40:19.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc771c20700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc76c0ccd40 0x7fc76c0cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:19.601 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc771c20700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc76c0ce390 0x7fc76c156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:19.601 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc771c20700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc76c0cf970 0x7fc76c0b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:19.601 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc771c20700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c0cb160 con 0x7fc76c0ce390 2023-06-23T00:40:19.601 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc771c20700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c15be90 con 0x7fc76c0ccd40 2023-06-23T00:40:19.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc771c20700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c15c170 con 0x7fc76c0cf970 2023-06-23T00:40:19.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc76affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc76c0ce390 0x7fc76c156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:19.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc76affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc76c0ce390 0x7fc76c156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41000/0 (socket says 172.21.15.64:41000) 2023-06-23T00:40:19.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc76affd700 1 -- 172.21.15.64:0/3661979096 learned_addr learned my addr 172.21.15.64:0/3661979096 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:19.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc76b7fe700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc76c0ccd40 0x7fc76c0cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:19.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc76bfff700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc76c0cf970 0x7fc76c0b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:19.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc76affd700 1 -- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc76c0cf970 msgr2=0x7fc76c0b4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:19.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.591+0000 7fc76affd700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc76c0cf970 0x7fc76c0b4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.592+0000 7fc76affd700 1 -- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc76c0ccd40 msgr2=0x7fc76c0cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:19.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.592+0000 7fc76affd700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc76c0ccd40 0x7fc76c0cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:19.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.592+0000 7fc76affd700 1 -- 172.21.15.64:0/3661979096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7600094f0 con 0x7fc76c0ce390 2023-06-23T00:40:19.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.592+0000 7fc76affd700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc76c0ce390 0x7fc76c156d60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc76c069cd0 tx=0x7fc760017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:19.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.592+0000 7fc7537fe700 1 -- 172.21.15.64:0/3661979096 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760011e40 con 0x7fc76c0ce390 2023-06-23T00:40:19.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.592+0000 7fc7537fe700 1 -- 172.21.15.64:0/3661979096 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76001a450 con 0x7fc76c0ce390 2023-06-23T00:40:19.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.592+0000 7fc771c20700 1 -- 172.21.15.64:0/3661979096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76c15c450 con 0x7fc76c0ce390 2023-06-23T00:40:19.604 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.592+0000 7fc7537fe700 1 -- 172.21.15.64:0/3661979096 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76001a810 con 0x7fc76c0ce390 2023-06-23T00:40:19.604 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.592+0000 7fc771c20700 1 -- 172.21.15.64:0/3661979096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76c15ca10 con 0x7fc76c0ce390 2023-06-23T00:40:19.605 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.593+0000 7fc771c20700 1 -- 172.21.15.64:0/3661979096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76c04c1b0 con 0x7fc76c0ce390 2023-06-23T00:40:19.605 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.593+0000 7fc7537fe700 1 -- 172.21.15.64:0/3661979096 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fc76001abc0 con 0x7fc76c0ce390 2023-06-23T00:40:19.605 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.594+0000 7fc7537fe700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fc758066e90 0x7fc758069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:19.605 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.594+0000 7fc7537fe700 1 -- 172.21.15.64:0/3661979096 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6777+0+0 (secure 0 0 0) 0x7fc7600862d0 con 0x7fc76c0ce390 2023-06-23T00:40:19.605 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.594+0000 7fc76b7fe700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fc758066e90 0x7fc758069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:19.605 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.594+0000 7fc76b7fe700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fc758066e90 0x7fc758069340 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc75c006fd0 tx=0x7fc75c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:19.606 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.598+0000 7fc7537fe700 1 -- 172.21.15.64:0/3661979096 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fc7600531d0 con 0x7fc76c0ce390 2023-06-23T00:40:19.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:19.790+0000 7fc771c20700 1 -- 172.21.15.64:0/3661979096 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fc76c0cf170 con 0x7fc76c0ce390 2023-06-23T00:40:20.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.437+0000 7fc7537fe700 1 -- 172.21.15.64:0/3661979096 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 5 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc7600567f0 con 0x7fc76c0ce390 2023-06-23T00:40:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.448+0000 7fc771c20700 1 -- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fc758066e90 msgr2=0x7fc758069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:20.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.448+0000 7fc771c20700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fc758066e90 0x7fc758069340 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc75c006fd0 tx=0x7fc75c008040 comp rx=0 tx=0).stop 2023-06-23T00:40:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.448+0000 7fc771c20700 1 -- 172.21.15.64:0/3661979096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc76c0ce390 msgr2=0x7fc76c156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:20.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.448+0000 7fc771c20700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc76c0ce390 0x7fc76c156d60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc76c069cd0 tx=0x7fc760017000 comp rx=0 tx=0).stop 2023-06-23T00:40:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.449+0000 7fc771c20700 1 -- 172.21.15.64:0/3661979096 shutdown_connections 2023-06-23T00:40:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.449+0000 7fc771c20700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7fc758066e90 0x7fc758069340 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.449+0000 7fc771c20700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc76c0cf970 0x7fc76c0b4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.449+0000 7fc771c20700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc76c0ccd40 0x7fc76c0cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.449+0000 7fc771c20700 1 --2- 172.21.15.64:0/3661979096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc76c0ce390 0x7fc76c156d60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.449+0000 7fc771c20700 1 -- 172.21.15.64:0/3661979096 >> 172.21.15.64:0/3661979096 conn(0x7fc76c087c70 msgr2=0x7fc76c0c6360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:20.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.450+0000 7fc771c20700 1 -- 172.21.15.64:0/3661979096 shutdown_connections 2023-06-23T00:40:20.455 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.450+0000 7fc771c20700 1 -- 172.21.15.64:0/3661979096 wait complete. 2023-06-23T00:40:20.465 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 5 allow_ec_overwrites to true 2023-06-23T00:40:20.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-06-23T00:40:20.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.604+0000 7feb8daef700 1 Processor -- start 2023-06-23T00:40:20.607 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.604+0000 7feb8daef700 1 -- start start 2023-06-23T00:40:20.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb8daef700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7feb8815d700 0x7feb8815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:20.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb8daef700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feb8815c4b0 0x7feb8815c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:20.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb8daef700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8815e0e0 con 0x7feb8815d330 2023-06-23T00:40:20.613 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb8daef700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8806a530 con 0x7feb8815d700 2023-06-23T00:40:20.613 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb8daef700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8806adf0 con 0x7feb8815c4b0 2023-06-23T00:40:20.613 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb87fff700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7feb8815d700 0x7feb8815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:20.613 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb8caed700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7feb8815d330 0x7feb880b21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43830/0 (socket says 172.21.15.64:43830) 2023-06-23T00:40:20.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb8caed700 1 -- 172.21.15.64:0/1158954131 learned_addr learned my addr 172.21.15.64:0/1158954131 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:20.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb87fff700 1 -- 172.21.15.64:0/1158954131 >> v2:172.21.15.86:3300/0 conn(0x7feb8815c4b0 msgr2=0x7feb8815c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:20.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb87fff700 1 --2- 172.21.15.64:0/1158954131 >> v2:172.21.15.86:3300/0 conn(0x7feb8815c4b0 0x7feb8815c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb87fff700 1 -- 172.21.15.64:0/1158954131 >> v1:172.21.15.64:6789/0 conn(0x7feb8815d330 legacy=0x7feb880b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:20.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb87fff700 1 -- 172.21.15.64:0/1158954131 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8806a7a0 con 0x7feb8815d700 2023-06-23T00:40:20.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.605+0000 7feb877fe700 1 -- 172.21.15.64:0/1158954131 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536826067 0 0) 0x7feb8815e0e0 con 0x7feb8815d330 2023-06-23T00:40:20.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb87fff700 1 --2- 172.21.15.64:0/1158954131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb8815d700 0x7feb8815dba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb880cca70 tx=0x7feb7c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249017e04ddf2f8d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:20.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb877fe700 1 -- 172.21.15.64:0/1158954131 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c00b6b0 con 0x7feb8815d700 2023-06-23T00:40:20.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb877fe700 1 -- 172.21.15.64:0/1158954131 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7c00b810 con 0x7feb8815d700 2023-06-23T00:40:20.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb877fe700 1 -- 172.21.15.64:0/1158954131 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c00bbb0 con 0x7feb8815d700 2023-06-23T00:40:20.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb8daef700 1 -- 172.21.15.64:0/1158954131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb8815d700 msgr2=0x7feb8815dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:20.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb8daef700 1 --2- 172.21.15.64:0/1158954131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb8815d700 0x7feb8815dba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb880cca70 tx=0x7feb7c0091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb8daef700 1 -- 172.21.15.64:0/1158954131 shutdown_connections 2023-06-23T00:40:20.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb8daef700 1 --2- 172.21.15.64:0/1158954131 >> v2:172.21.15.86:3300/0 conn(0x7feb8815c4b0 0x7feb8815c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb8daef700 1 --2- 172.21.15.64:0/1158954131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb8815d700 0x7feb8815dba0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb8daef700 1 -- 172.21.15.64:0/1158954131 >> 172.21.15.64:0/1158954131 conn(0x7feb880d0b30 msgr2=0x7feb880ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:20.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb8daef700 1 -- 172.21.15.64:0/1158954131 shutdown_connections 2023-06-23T00:40:20.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.606+0000 7feb8daef700 1 -- 172.21.15.64:0/1158954131 wait complete. 2023-06-23T00:40:20.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.607+0000 7feb8daef700 1 Processor -- start 2023-06-23T00:40:20.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb8daef700 1 -- start start 2023-06-23T00:40:20.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb8daef700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb8815c4b0 0x7feb880c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:20.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb8daef700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb8815d330 0x7feb8815b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:20.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb8daef700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb8815d700 0x7feb880c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:20.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb8daef700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb880c8760 con 0x7feb8815d700 2023-06-23T00:40:20.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb8daef700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb880c9130 con 0x7feb8815c4b0 2023-06-23T00:40:20.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb8daef700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb880c7b50 con 0x7feb8815d330 2023-06-23T00:40:20.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb8d2ee700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb8815d700 0x7feb880c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:20.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb8d2ee700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb8815d700 0x7feb880c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41018/0 (socket says 172.21.15.64:41018) 2023-06-23T00:40:20.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb8d2ee700 1 -- 172.21.15.64:0/2259816591 learned_addr learned my addr 172.21.15.64:0/2259816591 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:20.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb8caed700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb8815c4b0 0x7feb880c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:20.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.608+0000 7feb87fff700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb8815d330 0x7feb8815b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:20.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.609+0000 7feb8d2ee700 1 -- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb8815d330 msgr2=0x7feb8815b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:20.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.609+0000 7feb8d2ee700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb8815d330 0x7feb8815b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.609+0000 7feb8d2ee700 1 -- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb8815c4b0 msgr2=0x7feb880c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:20.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.609+0000 7feb8d2ee700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb8815c4b0 0x7feb880c8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.609+0000 7feb8d2ee700 1 -- 172.21.15.64:0/2259816591 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c009420 con 0x7feb8815d700 2023-06-23T00:40:20.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.609+0000 7feb8d2ee700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb8815d700 0x7feb880c70d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feb8000bc30 tx=0x7feb80009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:20.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.609+0000 7feb84ff9700 1 -- 172.21.15.64:0/2259816591 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800092e0 con 0x7feb8815d700 2023-06-23T00:40:20.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.609+0000 7feb84ff9700 1 -- 172.21.15.64:0/2259816591 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8000c630 con 0x7feb8815d700 2023-06-23T00:40:20.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.609+0000 7feb8daef700 1 -- 172.21.15.64:0/2259816591 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb880c7e30 con 0x7feb8815d700 2023-06-23T00:40:20.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.609+0000 7feb84ff9700 1 -- 172.21.15.64:0/2259816591 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8000ca10 con 0x7feb8815d700 2023-06-23T00:40:20.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.609+0000 7feb8daef700 1 -- 172.21.15.64:0/2259816591 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb880c83f0 con 0x7feb8815d700 2023-06-23T00:40:20.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.610+0000 7feb8daef700 1 -- 172.21.15.64:0/2259816591 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8815cfb0 con 0x7feb8815d700 2023-06-23T00:40:20.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.610+0000 7feb84ff9700 1 -- 172.21.15.64:0/2259816591 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7feb80012070 con 0x7feb8815d700 2023-06-23T00:40:20.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.611+0000 7feb84ff9700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7feb74066e90 0x7feb74069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:20.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.611+0000 7feb84ff9700 1 -- 172.21.15.64:0/2259816591 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6777+0+0 (secure 0 0 0) 0x7feb80084390 con 0x7feb8815d700 2023-06-23T00:40:20.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.611+0000 7feb8caed700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7feb74066e90 0x7feb74069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:20.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.611+0000 7feb8caed700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7feb74066e90 0x7feb74069340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb78007fd0 tx=0x7feb78007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:20.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.615+0000 7feb84ff9700 1 -- 172.21.15.64:0/2259816591 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7feb80051290 con 0x7feb8815d700 2023-06-23T00:40:20.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.856+0000 7feb8daef700 1 -- 172.21.15.64:0/2259816591 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7feb8804c1b0 con 0x7feb8815d700 2023-06-23T00:40:20.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.862+0000 7feb84ff9700 1 -- 172.21.15.64:0/2259816591 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 5 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7feb800548b0 con 0x7feb8815d700 2023-06-23T00:40:20.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 -- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7feb74066e90 msgr2=0x7feb74069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:20.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7feb74066e90 0x7feb74069340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feb78007fd0 tx=0x7feb78007d30 comp rx=0 tx=0).stop 2023-06-23T00:40:20.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 -- 172.21.15.64:0/2259816591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb8815d700 msgr2=0x7feb880c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:20.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb8815d700 0x7feb880c70d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feb8000bc30 tx=0x7feb80009840 comp rx=0 tx=0).stop 2023-06-23T00:40:20.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 -- 172.21.15.64:0/2259816591 shutdown_connections 2023-06-23T00:40:20.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7feb74066e90 0x7feb74069340 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb8815d700 0x7feb880c70d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb8815c4b0 0x7feb880c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 --2- 172.21.15.64:0/2259816591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb8815d330 0x7feb8815b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:20.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 -- 172.21.15.64:0/2259816591 >> 172.21.15.64:0/2259816591 conn(0x7feb880d0b30 msgr2=0x7feb880afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:20.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 -- 172.21.15.64:0/2259816591 shutdown_connections 2023-06-23T00:40:20.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.864+0000 7feb8daef700 1 -- 172.21.15.64:0/2259816591 wait complete. 2023-06-23T00:40:20.869 INFO:teuthology.orchestra.run.smithi064.stderr:added data pool 5 to fsmap 2023-06-23T00:40:20.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T00:40:20.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.995+0000 7f29a259b700 1 Processor -- start 2023-06-23T00:40:20.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.995+0000 7f29a259b700 1 -- start start 2023-06-23T00:40:20.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.995+0000 7f29a259b700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f299c0d04f0 0x7f299c15a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:20.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.995+0000 7f29a259b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f299c1570c0 0x7f299c0cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:20.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f29a259b700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299c15aaa0 con 0x7f299c0d08c0 2023-06-23T00:40:20.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f29a259b700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f299c0ce5c0 con 0x7f299c0d04f0 2023-06-23T00:40:20.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f29a259b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0ccf70 con 0x7f299c1570c0 2023-06-23T00:40:20.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299bfff700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f299c0d04f0 0x7f299c15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299b7fe700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f299c0d08c0 0x7f299c1569b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43834/0 (socket says 172.21.15.64:43834) 2023-06-23T00:40:21.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299b7fe700 1 -- 172.21.15.64:0/1623890061 learned_addr learned my addr 172.21.15.64:0/1623890061 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:21.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f29a0b38700 1 --2- 172.21.15.64:0/1623890061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f299c1570c0 0x7f299c0cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299bfff700 1 -- 172.21.15.64:0/1623890061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f299c1570c0 msgr2=0x7f299c0cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:21.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299bfff700 1 --2- 172.21.15.64:0/1623890061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f299c1570c0 0x7f299c0cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299bfff700 1 -- 172.21.15.64:0/1623890061 >> v1:172.21.15.64:6789/0 conn(0x7f299c0d08c0 legacy=0x7f299c1569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299affd700 1 -- 172.21.15.64:0/1623890061 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006138397 0 0) 0x7f299c15aaa0 con 0x7f299c0d08c0 2023-06-23T00:40:21.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299bfff700 1 -- 172.21.15.64:0/1623890061 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c0cd1d0 con 0x7f299c0d04f0 2023-06-23T00:40:21.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299bfff700 1 --2- 172.21.15.64:0/1623890061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f299c0d04f0 0x7f299c15a410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f298c0098e0 tx=0x7f298c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8613098482c08d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:21.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299affd700 1 -- 172.21.15.64:0/1623890061 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c009e80 con 0x7f299c0d04f0 2023-06-23T00:40:21.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299affd700 1 -- 172.21.15.64:0/1623890061 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c00b860 con 0x7f299c0d04f0 2023-06-23T00:40:21.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.996+0000 7f299affd700 1 -- 172.21.15.64:0/1623890061 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c00bbe0 con 0x7f299c0d04f0 2023-06-23T00:40:21.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.997+0000 7f29a259b700 1 -- 172.21.15.64:0/1623890061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f299c0d04f0 msgr2=0x7f299c15a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.997+0000 7f29a259b700 1 --2- 172.21.15.64:0/1623890061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f299c0d04f0 0x7f299c15a410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f298c0098e0 tx=0x7f298c0091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.997+0000 7f29a259b700 1 -- 172.21.15.64:0/1623890061 shutdown_connections 2023-06-23T00:40:21.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.997+0000 7f29a259b700 1 --2- 172.21.15.64:0/1623890061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f299c0d04f0 0x7f299c15a410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.997+0000 7f29a259b700 1 --2- 172.21.15.64:0/1623890061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f299c1570c0 0x7f299c0cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.997+0000 7f29a259b700 1 -- 172.21.15.64:0/1623890061 >> 172.21.15.64:0/1623890061 conn(0x7f299c06a060 msgr2=0x7f299c06a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:21.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.997+0000 7f29a259b700 1 -- 172.21.15.64:0/1623890061 shutdown_connections 2023-06-23T00:40:21.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.997+0000 7f29a259b700 1 -- 172.21.15.64:0/1623890061 wait complete. 2023-06-23T00:40:21.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.998+0000 7f29a259b700 1 Processor -- start 2023-06-23T00:40:21.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.998+0000 7f29a259b700 1 -- start start 2023-06-23T00:40:21.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f29a259b700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f299c0d04f0 0x7f299c14e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f29a259b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f299c0d08c0 0x7f299c15dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f29a259b700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f299c1570c0 0x7f299c154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f29a259b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0b3080 con 0x7f299c0d08c0 2023-06-23T00:40:21.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f29a259b700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0b3210 con 0x7f299c0d04f0 2023-06-23T00:40:21.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f29a259b700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299c15e550 con 0x7f299c1570c0 2023-06-23T00:40:21.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f299b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f299c0d08c0 0x7f299c15dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f299b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f299c0d08c0 0x7f299c15dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41036/0 (socket says 172.21.15.64:41036) 2023-06-23T00:40:21.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f299b7fe700 1 -- 172.21.15.64:0/3043279242 learned_addr learned my addr 172.21.15.64:0/3043279242 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:21.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f29a0b38700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f299c1570c0 0x7f299c154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f299bfff700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f299c0d04f0 0x7f299c14e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f299b7fe700 1 -- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f299c1570c0 msgr2=0x7f299c154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f299b7fe700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f299c1570c0 0x7f299c154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f299b7fe700 1 -- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f299c0d04f0 msgr2=0x7f299c14e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f299b7fe700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f299c0d04f0 0x7f299c14e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f299b7fe700 1 -- 172.21.15.64:0/3043279242 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c009420 con 0x7f299c0d08c0 2023-06-23T00:40:21.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f29a0b38700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f299c1570c0 0x7f299c154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T00:40:21.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:20.999+0000 7f299bfff700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f299c0d04f0 0x7f299c14e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T00:40:21.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.000+0000 7f299b7fe700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f299c0d08c0 0x7f299c15dab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f299000c810 tx=0x7f2990007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:21.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.000+0000 7f2983fff700 1 -- 172.21.15.64:0/3043279242 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2990005b30 con 0x7f299c0d08c0 2023-06-23T00:40:21.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.000+0000 7f2983fff700 1 -- 172.21.15.64:0/3043279242 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2990005c90 con 0x7f299c0d08c0 2023-06-23T00:40:21.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.000+0000 7f29a259b700 1 -- 172.21.15.64:0/3043279242 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299c15e830 con 0x7f299c0d08c0 2023-06-23T00:40:21.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.000+0000 7f2983fff700 1 -- 172.21.15.64:0/3043279242 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2990007300 con 0x7f299c0d08c0 2023-06-23T00:40:21.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.000+0000 7f29a259b700 1 -- 172.21.15.64:0/3043279242 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299c15ee70 con 0x7f299c0d08c0 2023-06-23T00:40:21.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.001+0000 7f29a259b700 1 -- 172.21.15.64:0/3043279242 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299c04c1b0 con 0x7f299c0d08c0 2023-06-23T00:40:21.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.001+0000 7f2983fff700 1 -- 172.21.15.64:0/3043279242 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f299001b400 con 0x7f299c0d08c0 2023-06-23T00:40:21.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.001+0000 7f2983fff700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2988066e90 0x7f2988069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.002+0000 7f2983fff700 1 -- 172.21.15.64:0/3043279242 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f29900844c0 con 0x7f299c0d08c0 2023-06-23T00:40:21.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.002+0000 7f299bfff700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2988066e90 0x7f2988069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.002+0000 7f299bfff700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2988066e90 0x7f2988069340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f298c0091e0 tx=0x7f298c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:21.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.006+0000 7f2983fff700 1 -- 172.21.15.64:0/3043279242 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f2990051300 con 0x7f299c0d08c0 2023-06-23T00:40:21.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.248+0000 7f29a259b700 1 -- 172.21.15.64:0/3043279242 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f299c0b3350 con 0x7f299c0d08c0 2023-06-23T00:40:21.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.249+0000 7f2983fff700 1 -- 172.21.15.64:0/3043279242 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4613 (secure 0 0 0) 0x7f2990054920 con 0x7f299c0d08c0 2023-06-23T00:40:21.251 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:21.252 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6834/1840968204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6833","nonce":1840968204},{"type":"v1","addr":"172.21.15.64:6834","nonce":1840968204}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/1141461387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1141461387},{"type":"v1","addr":"172.21.15.86:6835","nonce":1141461387}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1370273364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1370273364},{"type":"v1","addr":"172.21.15.64:6835","nonce":1370273364}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-23T00:40:16.414564+0000","modified":"2023-06-23T00:40:20.857359+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/895538016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":895538016},{"type":"v1","addr":"172.21.15.86:6837","nonce":895538016}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-06-23T00:40:21.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 -- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2988066e90 msgr2=0x7f2988069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2988066e90 0x7f2988069340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f298c0091e0 tx=0x7f298c017000 comp rx=0 tx=0).stop 2023-06-23T00:40:21.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 -- 172.21.15.64:0/3043279242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f299c0d08c0 msgr2=0x7f299c15dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f299c0d08c0 0x7f299c15dab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f299000c810 tx=0x7f2990007df0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 -- 172.21.15.64:0/3043279242 shutdown_connections 2023-06-23T00:40:21.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f2988066e90 0x7f2988069340 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f299c1570c0 0x7f299c154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f299c0d04f0 0x7f299c14e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 --2- 172.21.15.64:0/3043279242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f299c0d08c0 0x7f299c15dab0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 -- 172.21.15.64:0/3043279242 >> 172.21.15.64:0/3043279242 conn(0x7f299c06a060 msgr2=0x7f299c156780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:21.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 -- 172.21.15.64:0/3043279242 shutdown_connections 2023-06-23T00:40:21.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.251+0000 7f29a259b700 1 -- 172.21.15.64:0/3043279242 wait complete. 2023-06-23T00:40:21.256 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 7 2023-06-23T00:40:21.265 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T00:40:21.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.381+0000 7f33e3af6700 1 Processor -- start 2023-06-23T00:40:21.436 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.382+0000 7f33e3af6700 1 -- start start 2023-06-23T00:40:21.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.382+0000 7f33e3af6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc15c720 0x7f33dc15a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.382+0000 7f33e3af6700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33dc15aaf0 0x7f33dc0d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e3af6700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33dc15a8a0 con 0x7f33dc069a50 2023-06-23T00:40:21.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e3af6700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33dc0cea50 con 0x7f33dc15aaf0 2023-06-23T00:40:21.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e3af6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc06a010 con 0x7f33dc15c720 2023-06-23T00:40:21.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e1892700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc15c720 0x7f33dc15a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e1091700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33dc15aaf0 0x7f33dc0d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e1091700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33dc15aaf0 0x7f33dc0d00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.64:38302/0 (socket says 172.21.15.64:38302) 2023-06-23T00:40:21.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e1892700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc15c720 0x7f33dc15a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.64:48352/0 (socket says 172.21.15.64:48352) 2023-06-23T00:40:21.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e1091700 1 -- 172.21.15.64:0/276444236 learned_addr learned my addr 172.21.15.64:0/276444236 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:21.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e2093700 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f33dc069a50 0x7f33dc0cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.64:43734/0 (socket says 172.21.15.64:43734) 2023-06-23T00:40:21.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e1892700 1 -- 172.21.15.64:0/276444236 >> v2:172.21.15.86:3300/0 conn(0x7f33dc15aaf0 msgr2=0x7f33dc0d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:21.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e1892700 1 --2- 172.21.15.64:0/276444236 >> v2:172.21.15.86:3300/0 conn(0x7f33dc15aaf0 0x7f33dc0d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e1892700 1 -- 172.21.15.64:0/276444236 >> v1:172.21.15.86:6789/0 conn(0x7f33dc069a50 legacy=0x7f33dc0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e1892700 1 -- 172.21.15.64:0/276444236 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc06a2f0 con 0x7f33dc15c720 2023-06-23T00:40:21.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e1892700 1 --2- 172.21.15.64:0/276444236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc15c720 0x7f33dc15a360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f33d40098e0 tx=0x7f33d40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71fc14337135ee8d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:21.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.383+0000 7f33e0890700 1 -- 172.21.15.64:0/276444236 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d400b640 con 0x7f33dc15c720 2023-06-23T00:40:21.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.384+0000 7f33e0890700 1 -- 172.21.15.64:0/276444236 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d400b7a0 con 0x7f33dc15c720 2023-06-23T00:40:21.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.384+0000 7f33e0890700 1 -- 172.21.15.64:0/276444236 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d400bb40 con 0x7f33dc15c720 2023-06-23T00:40:21.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.384+0000 7f33e3af6700 1 -- 172.21.15.64:0/276444236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc15c720 msgr2=0x7f33dc15a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.384+0000 7f33e3af6700 1 --2- 172.21.15.64:0/276444236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc15c720 0x7f33dc15a360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f33d40098e0 tx=0x7f33d40091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.384+0000 7f33e3af6700 1 -- 172.21.15.64:0/276444236 shutdown_connections 2023-06-23T00:40:21.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.384+0000 7f33e3af6700 1 --2- 172.21.15.64:0/276444236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc15c720 0x7f33dc15a360 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.384+0000 7f33e3af6700 1 --2- 172.21.15.64:0/276444236 >> v2:172.21.15.86:3300/0 conn(0x7f33dc15aaf0 0x7f33dc0d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.384+0000 7f33e3af6700 1 -- 172.21.15.64:0/276444236 >> 172.21.15.64:0/276444236 conn(0x7f33dc068510 msgr2=0x7f33dc068910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:21.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.384+0000 7f33e3af6700 1 -- 172.21.15.64:0/276444236 shutdown_connections 2023-06-23T00:40:21.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.384+0000 7f33e3af6700 1 -- 172.21.15.64:0/276444236 wait complete. 2023-06-23T00:40:21.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.385+0000 7f33e3af6700 1 Processor -- start 2023-06-23T00:40:21.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.385+0000 7f33e3af6700 1 -- start start 2023-06-23T00:40:21.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e3af6700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc069a50 0x7f33dc151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e3af6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33dc15aaf0 0x7f33dc14fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e3af6700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33dc15c720 0x7f33dc13f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e3af6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc14e100 con 0x7f33dc15aaf0 2023-06-23T00:40:21.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e3af6700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc14d7d0 con 0x7f33dc15c720 2023-06-23T00:40:21.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e3af6700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc14dab0 con 0x7f33dc069a50 2023-06-23T00:40:21.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e1091700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33dc15aaf0 0x7f33dc14fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e1091700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33dc15aaf0 0x7f33dc14fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41044/0 (socket says 172.21.15.64:41044) 2023-06-23T00:40:21.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e1091700 1 -- 172.21.15.64:0/3126063144 learned_addr learned my addr 172.21.15.64:0/3126063144 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:21.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e2093700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33dc15c720 0x7f33dc13f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e1892700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc069a50 0x7f33dc151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e1091700 1 -- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc069a50 msgr2=0x7f33dc151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e1091700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc069a50 0x7f33dc151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e1091700 1 -- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33dc15c720 msgr2=0x7f33dc13f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e1091700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33dc15c720 0x7f33dc13f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e1091700 1 -- 172.21.15.64:0/3126063144 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d4009420 con 0x7f33dc15aaf0 2023-06-23T00:40:21.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.386+0000 7f33e1892700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc069a50 0x7f33dc151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T00:40:21.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.387+0000 7f33e1091700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33dc15aaf0 0x7f33dc14fab0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f33d000ea50 tx=0x7f33d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:21.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.387+0000 7f33d9ffb700 1 -- 172.21.15.64:0/3126063144 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d000eee0 con 0x7f33dc15aaf0 2023-06-23T00:40:21.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.387+0000 7f33d9ffb700 1 -- 172.21.15.64:0/3126063144 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d000c850 con 0x7f33dc15aaf0 2023-06-23T00:40:21.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.387+0000 7f33e3af6700 1 -- 172.21.15.64:0/3126063144 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33dc14dd90 con 0x7f33dc15aaf0 2023-06-23T00:40:21.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.387+0000 7f33d9ffb700 1 -- 172.21.15.64:0/3126063144 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d000cbd0 con 0x7f33dc15aaf0 2023-06-23T00:40:21.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.387+0000 7f33e3af6700 1 -- 172.21.15.64:0/3126063144 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33dc14fff0 con 0x7f33dc15aaf0 2023-06-23T00:40:21.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.388+0000 7f33e3af6700 1 -- 172.21.15.64:0/3126063144 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33dc14e2b0 con 0x7f33dc15aaf0 2023-06-23T00:40:21.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.388+0000 7f33d9ffb700 1 -- 172.21.15.64:0/3126063144 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f33d0013470 con 0x7f33dc15aaf0 2023-06-23T00:40:21.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.389+0000 7f33d9ffb700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c8066e90 0x7f33c8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.389+0000 7f33d9ffb700 1 -- 172.21.15.64:0/3126063144 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f33d0084a50 con 0x7f33dc15aaf0 2023-06-23T00:40:21.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.389+0000 7f33e1892700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c8066e90 0x7f33c8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.389+0000 7f33e1892700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c8066e90 0x7f33c8069340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f33d40093f0 tx=0x7f33d4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:21.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.393+0000 7f33d9ffb700 1 -- 172.21.15.64:0/3126063144 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f33dc14e2b0 con 0x7f33dc15aaf0 2023-06-23T00:40:21.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.586+0000 7f33e3af6700 1 -- 172.21.15.64:0/3126063144 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33dc04c1b0 con 0x7f33dc15aaf0 2023-06-23T00:40:21.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.587+0000 7f33d9ffb700 1 -- 172.21.15.64:0/3126063144 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f33d00553c0 con 0x7f33dc15aaf0 2023-06-23T00:40:21.590 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:21.590 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":24,"fsid":"72efde62-4c91-4277-87ee-356e50bd4cf5","created":"2023-06-23T00:39:58.078232+0000","modified":"2023-06-23T00:40:20.857067+0000","last_up_change":"2023-06-23T00:40:08.853479+0000","last_in_change":"2023-06-23T00:40:00.493656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-23T00:40:09.294447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-23T00:40:09.719568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T00:40:14.182408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T00:40:15.395198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-23T00:40:18.457653+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":88027},{"type":"v1","addr":"172.21.15.64:6801","nonce":88027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":88027},{"type":"v1","addr":"172.21.15.64:6803","nonce":88027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":88027},{"type":"v1","addr":"172.21.15.64:6807","nonce":88027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":88027},{"type":"v1","addr":"172.21.15.64:6805","nonce":88027}]},"public_addr":"172.21.15.64:6801/88027","cluster_addr":"172.21.15.64:6803/88027","heartbeat_back_addr":"172.21.15.64:6807/88027","heartbeat_front_addr":"172.21.15.64:6805/88027","state":["exists","up"]},{"osd":1,"uuid":"f0698c9d-c67a-42ac-8d36-838836c514d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":88062},{"type":"v1","addr":"172.21.15.64:6825","nonce":88062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":88062},{"type":"v1","addr":"172.21.15.64:6827","nonce":88062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":88062},{"type":"v1","addr":"172.21.15.64:6831","nonce":88062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":88062},{"type":"v1","addr":"172.21.15.64:6829","nonce":88062}]},"public_addr":"172.21.15.64:6825/88062","cluster_addr":"172.21.15.64:6827/88062","heartbeat_back_addr":"172.21.15.64:6831/88062","heartbeat_front_addr":"172.21.15.64:6829/88062","state":["exists","up"]},{"osd":2,"uuid":"a5ece869-9da7-40e8-8f1d-bbaf67bb2187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":88045},{"type":"v1","addr":"172.21.15.64:6817","nonce":88045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":88045},{"type":"v1","addr":"172.21.15.64:6819","nonce":88045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":88045},{"type":"v1","addr":"172.21.15.64:6823","nonce":88045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":88045},{"type":"v1","addr":"172.21.15.64:6821","nonce":88045}]},"public_addr":"172.21.15.64:6817/88045","cluster_addr":"172.21.15.64:6819/88045","heartbeat_back_addr":"172.21.15.64:6823/88045","heartbeat_front_addr":"172.21.15.64:6821/88045","state":["exists","up"]},{"osd":3,"uuid":"21067fac-663b-4ecf-ae67-bf235161638e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":88044},{"type":"v1","addr":"172.21.15.64:6809","nonce":88044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":88044},{"type":"v1","addr":"172.21.15.64:6811","nonce":88044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":88044},{"type":"v1","addr":"172.21.15.64:6815","nonce":88044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":88044},{"type":"v1","addr":"172.21.15.64:6813","nonce":88044}]},"public_addr":"172.21.15.64:6809/88044","cluster_addr":"172.21.15.64:6811/88044","heartbeat_back_addr":"172.21.15.64:6815/88044","heartbeat_front_addr":"172.21.15.64:6813/88044","state":["exists","up"]},{"osd":4,"uuid":"6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":118144},{"type":"v1","addr":"172.21.15.86:6801","nonce":118144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":118144},{"type":"v1","addr":"172.21.15.86:6803","nonce":118144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":118144},{"type":"v1","addr":"172.21.15.86:6807","nonce":118144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":118144},{"type":"v1","addr":"172.21.15.86:6805","nonce":118144}]},"public_addr":"172.21.15.86:6801/118144","cluster_addr":"172.21.15.86:6803/118144","heartbeat_back_addr":"172.21.15.86:6807/118144","heartbeat_front_addr":"172.21.15.86:6805/118144","state":["exists","up"]},{"osd":5,"uuid":"0e987a73-8b36-4f26-94fd-f0135d6062c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":118169},{"type":"v1","addr":"172.21.15.86:6817","nonce":118169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":118169},{"type":"v1","addr":"172.21.15.86:6819","nonce":118169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":118169},{"type":"v1","addr":"172.21.15.86:6823","nonce":118169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":118169},{"type":"v1","addr":"172.21.15.86:6821","nonce":118169}]},"public_addr":"172.21.15.86:6817/118169","cluster_addr":"172.21.15.86:6819/118169","heartbeat_back_addr":"172.21.15.86:6823/118169","heartbeat_front_addr":"172.21.15.86:6821/118169","state":["exists","up"]},{"osd":6,"uuid":"7e40bfd5-0f9f-448c-8ccf-2f44bf82a622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":118175},{"type":"v1","addr":"172.21.15.86:6825","nonce":118175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":118175},{"type":"v1","addr":"172.21.15.86:6827","nonce":118175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":118175},{"type":"v1","addr":"172.21.15.86:6831","nonce":118175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":118175},{"type":"v1","addr":"172.21.15.86:6829","nonce":118175}]},"public_addr":"172.21.15.86:6825/118175","cluster_addr":"172.21.15.86:6827/118175","heartbeat_back_addr":"172.21.15.86:6831/118175","heartbeat_front_addr":"172.21.15.86:6829/118175","state":["exists","up"]},{"osd":7,"uuid":"2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":118145},{"type":"v1","addr":"172.21.15.86:6809","nonce":118145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":118145},{"type":"v1","addr":"172.21.15.86:6811","nonce":118145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":118145},{"type":"v1","addr":"172.21.15.86:6815","nonce":118145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":118145},{"type":"v1","addr":"172.21.15.86:6813","nonce":118145}]},"public_addr":"172.21.15.86:6809/118145","cluster_addr":"172.21.15.86:6811/118145","heartbeat_back_addr":"172.21.15.86:6815/118145","heartbeat_front_addr":"172.21.15.86:6813/118145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.319777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.692469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.685180+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.648752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.461468+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.727692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.701726+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.703200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T00:40:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 -- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c8066e90 msgr2=0x7f33c8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c8066e90 0x7f33c8069340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f33d40093f0 tx=0x7f33d4017000 comp rx=0 tx=0).stop 2023-06-23T00:40:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 -- 172.21.15.64:0/3126063144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33dc15aaf0 msgr2=0x7f33dc14fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33dc15aaf0 0x7f33dc14fab0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f33d000ea50 tx=0x7f33d0009250 comp rx=0 tx=0).stop 2023-06-23T00:40:21.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 -- 172.21.15.64:0/3126063144 shutdown_connections 2023-06-23T00:40:21.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c8066e90 0x7f33c8069340 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33dc15c720 0x7f33dc13f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33dc069a50 0x7f33dc151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 --2- 172.21.15.64:0/3126063144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33dc15aaf0 0x7f33dc14fab0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 -- 172.21.15.64:0/3126063144 >> 172.21.15.64:0/3126063144 conn(0x7f33dc068510 msgr2=0x7f33dc157540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:21.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 -- 172.21.15.64:0/3126063144 shutdown_connections 2023-06-23T00:40:21.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.589+0000 7f33e3af6700 1 -- 172.21.15.64:0/3126063144 wait complete. 2023-06-23T00:40:21.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T00:40:21.725 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.723+0000 7f8234b38700 1 Processor -- start 2023-06-23T00:40:21.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.723+0000 7f8234b38700 1 -- start start 2023-06-23T00:40:21.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.724+0000 7f8234b38700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82300d04f0 0x7f823015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.724+0000 7f8234b38700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f82301570c0 0x7f82300cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.724+0000 7f8234b38700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823015aaa0 con 0x7f82300d08c0 2023-06-23T00:40:21.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.724+0000 7f8234b38700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82300ce5c0 con 0x7f82301570c0 2023-06-23T00:40:21.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.724+0000 7f8234b38700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82300ccf70 con 0x7f82300d04f0 2023-06-23T00:40:21.728 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.724+0000 7f822e59c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82300d04f0 0x7f823015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.728 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.724+0000 7f822ed9d700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f82301570c0 0x7f82300cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.724+0000 7f822e59c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82300d04f0 0x7f823015a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.64:48368/0 (socket says 172.21.15.64:48368) 2023-06-23T00:40:21.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.724+0000 7f822e59c700 1 -- 172.21.15.64:0/293675010 learned_addr learned my addr 172.21.15.64:0/293675010 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:21.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f822e59c700 1 -- 172.21.15.64:0/293675010 >> v2:172.21.15.86:3300/0 conn(0x7f82301570c0 msgr2=0x7f82300cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:21.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f822e59c700 1 --2- 172.21.15.64:0/293675010 >> v2:172.21.15.86:3300/0 conn(0x7f82301570c0 0x7f82300cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f822e59c700 1 -- 172.21.15.64:0/293675010 >> v1:172.21.15.86:6789/0 conn(0x7f82300d08c0 legacy=0x7f82301569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f822e59c700 1 -- 172.21.15.64:0/293675010 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82300cd1d0 con 0x7f82300d04f0 2023-06-23T00:40:21.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f822ed9d700 1 --2- 172.21.15.64:0/293675010 >> v2:172.21.15.86:3300/0 conn(0x7f82301570c0 0x7f82300cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:21.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f822e59c700 1 --2- 172.21.15.64:0/293675010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82300d04f0 0x7f823015a410 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f82200099e0 tx=0x7f8220009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ada96891e3a4aa2 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:21.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f822d59a700 1 -- 172.21.15.64:0/293675010 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822000b820 con 0x7f82300d04f0 2023-06-23T00:40:21.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f822d59a700 1 -- 172.21.15.64:0/293675010 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f822000b980 con 0x7f82300d04f0 2023-06-23T00:40:21.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f822d59a700 1 -- 172.21.15.64:0/293675010 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822000bd20 con 0x7f82300d04f0 2023-06-23T00:40:21.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f8234b38700 1 -- 172.21.15.64:0/293675010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82300d04f0 msgr2=0x7f823015a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f8234b38700 1 --2- 172.21.15.64:0/293675010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82300d04f0 0x7f823015a410 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f82200099e0 tx=0x7f8220009250 comp rx=0 tx=0).stop 2023-06-23T00:40:21.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f8234b38700 1 -- 172.21.15.64:0/293675010 shutdown_connections 2023-06-23T00:40:21.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f8234b38700 1 --2- 172.21.15.64:0/293675010 >> v2:172.21.15.86:3300/0 conn(0x7f82301570c0 0x7f82300cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f8234b38700 1 --2- 172.21.15.64:0/293675010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82300d04f0 0x7f823015a410 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.725+0000 7f8234b38700 1 -- 172.21.15.64:0/293675010 >> 172.21.15.64:0/293675010 conn(0x7f823006a060 msgr2=0x7f823006a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:21.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.726+0000 7f8234b38700 1 -- 172.21.15.64:0/293675010 shutdown_connections 2023-06-23T00:40:21.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.726+0000 7f8234b38700 1 -- 172.21.15.64:0/293675010 wait complete. 2023-06-23T00:40:21.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.727+0000 7f8234b38700 1 Processor -- start 2023-06-23T00:40:21.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.727+0000 7f8234b38700 1 -- start start 2023-06-23T00:40:21.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.727+0000 7f8234b38700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82300d04f0 0x7f8230148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.727+0000 7f8234b38700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82300d08c0 0x7f82300b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.727+0000 7f8234b38700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82301570c0 0x7f82300c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.727+0000 7f8234b38700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823016a960 con 0x7f82300d04f0 2023-06-23T00:40:21.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.727+0000 7f8234b38700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82300b0ec0 con 0x7f82300d08c0 2023-06-23T00:40:21.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.727+0000 7f8234b38700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8230149290 con 0x7f82301570c0 2023-06-23T00:40:21.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.727+0000 7f822e59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82300d04f0 0x7f8230148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.727+0000 7f822e59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82300d04f0 0x7f8230148d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41058/0 (socket says 172.21.15.64:41058) 2023-06-23T00:40:21.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.727+0000 7f822e59c700 1 -- 172.21.15.64:0/86906432 learned_addr learned my addr 172.21.15.64:0/86906432 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:21.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f822ed9d700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82301570c0 0x7f82300c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f822dd9b700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82300d08c0 0x7f82300b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f822e59c700 1 -- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82301570c0 msgr2=0x7f82300c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f822e59c700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82301570c0 0x7f82300c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f822e59c700 1 -- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82300d08c0 msgr2=0x7f82300b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f822e59c700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82300d08c0 0x7f82300b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f822e59c700 1 -- 172.21.15.64:0/86906432 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8228009a20 con 0x7f82300d04f0 2023-06-23T00:40:21.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f822dd9b700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82300d08c0 0x7f82300b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T00:40:21.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f822e59c700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82300d04f0 0x7f8230148d50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8220009290 tx=0x7f8220017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:21.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f82267fc700 1 -- 172.21.15.64:0/86906432 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8220024440 con 0x7f82300d04f0 2023-06-23T00:40:21.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f82267fc700 1 -- 172.21.15.64:0/86906432 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82200245e0 con 0x7f82300d04f0 2023-06-23T00:40:21.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f8234b38700 1 -- 172.21.15.64:0/86906432 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82200094f0 con 0x7f82300d04f0 2023-06-23T00:40:21.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f82267fc700 1 -- 172.21.15.64:0/86906432 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82200249a0 con 0x7f82300d04f0 2023-06-23T00:40:21.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.728+0000 7f8234b38700 1 -- 172.21.15.64:0/86906432 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82300b0590 con 0x7f82300d04f0 2023-06-23T00:40:21.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.729+0000 7f8234b38700 1 -- 172.21.15.64:0/86906432 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f823004c1b0 con 0x7f82300d04f0 2023-06-23T00:40:21.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.730+0000 7f82267fc700 1 -- 172.21.15.64:0/86906432 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f822001a420 con 0x7f82300d04f0 2023-06-23T00:40:21.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.730+0000 7f82267fc700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f8210066e90 0x7f8210069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:21.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.730+0000 7f82267fc700 1 -- 172.21.15.64:0/86906432 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f8220086550 con 0x7f82300d04f0 2023-06-23T00:40:21.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.730+0000 7f822dd9b700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f8210066e90 0x7f8210069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:21.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.731+0000 7f822dd9b700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f8210066e90 0x7f8210069340 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8218007fd0 tx=0x7f8218009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:21.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.734+0000 7f82267fc700 1 -- 172.21.15.64:0/86906432 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f82200534f0 con 0x7f82300d04f0 2023-06-23T00:40:21.933 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.931+0000 7f8234b38700 1 -- 172.21.15.64:0/86906432 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82300cd4b0 con 0x7f82300d04f0 2023-06-23T00:40:21.934 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.932+0000 7f82267fc700 1 -- 172.21.15.64:0/86906432 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f8220056b10 con 0x7f82300d04f0 2023-06-23T00:40:21.935 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:21.935 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":24,"fsid":"72efde62-4c91-4277-87ee-356e50bd4cf5","created":"2023-06-23T00:39:58.078232+0000","modified":"2023-06-23T00:40:20.857067+0000","last_up_change":"2023-06-23T00:40:08.853479+0000","last_in_change":"2023-06-23T00:40:00.493656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-23T00:40:09.294447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-23T00:40:09.719568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T00:40:14.182408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T00:40:15.395198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-23T00:40:18.457653+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":88027},{"type":"v1","addr":"172.21.15.64:6801","nonce":88027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":88027},{"type":"v1","addr":"172.21.15.64:6803","nonce":88027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":88027},{"type":"v1","addr":"172.21.15.64:6807","nonce":88027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":88027},{"type":"v1","addr":"172.21.15.64:6805","nonce":88027}]},"public_addr":"172.21.15.64:6801/88027","cluster_addr":"172.21.15.64:6803/88027","heartbeat_back_addr":"172.21.15.64:6807/88027","heartbeat_front_addr":"172.21.15.64:6805/88027","state":["exists","up"]},{"osd":1,"uuid":"f0698c9d-c67a-42ac-8d36-838836c514d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":88062},{"type":"v1","addr":"172.21.15.64:6825","nonce":88062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":88062},{"type":"v1","addr":"172.21.15.64:6827","nonce":88062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":88062},{"type":"v1","addr":"172.21.15.64:6831","nonce":88062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":88062},{"type":"v1","addr":"172.21.15.64:6829","nonce":88062}]},"public_addr":"172.21.15.64:6825/88062","cluster_addr":"172.21.15.64:6827/88062","heartbeat_back_addr":"172.21.15.64:6831/88062","heartbeat_front_addr":"172.21.15.64:6829/88062","state":["exists","up"]},{"osd":2,"uuid":"a5ece869-9da7-40e8-8f1d-bbaf67bb2187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":88045},{"type":"v1","addr":"172.21.15.64:6817","nonce":88045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":88045},{"type":"v1","addr":"172.21.15.64:6819","nonce":88045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":88045},{"type":"v1","addr":"172.21.15.64:6823","nonce":88045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":88045},{"type":"v1","addr":"172.21.15.64:6821","nonce":88045}]},"public_addr":"172.21.15.64:6817/88045","cluster_addr":"172.21.15.64:6819/88045","heartbeat_back_addr":"172.21.15.64:6823/88045","heartbeat_front_addr":"172.21.15.64:6821/88045","state":["exists","up"]},{"osd":3,"uuid":"21067fac-663b-4ecf-ae67-bf235161638e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":88044},{"type":"v1","addr":"172.21.15.64:6809","nonce":88044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":88044},{"type":"v1","addr":"172.21.15.64:6811","nonce":88044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":88044},{"type":"v1","addr":"172.21.15.64:6815","nonce":88044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":88044},{"type":"v1","addr":"172.21.15.64:6813","nonce":88044}]},"public_addr":"172.21.15.64:6809/88044","cluster_addr":"172.21.15.64:6811/88044","heartbeat_back_addr":"172.21.15.64:6815/88044","heartbeat_front_addr":"172.21.15.64:6813/88044","state":["exists","up"]},{"osd":4,"uuid":"6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":118144},{"type":"v1","addr":"172.21.15.86:6801","nonce":118144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":118144},{"type":"v1","addr":"172.21.15.86:6803","nonce":118144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":118144},{"type":"v1","addr":"172.21.15.86:6807","nonce":118144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":118144},{"type":"v1","addr":"172.21.15.86:6805","nonce":118144}]},"public_addr":"172.21.15.86:6801/118144","cluster_addr":"172.21.15.86:6803/118144","heartbeat_back_addr":"172.21.15.86:6807/118144","heartbeat_front_addr":"172.21.15.86:6805/118144","state":["exists","up"]},{"osd":5,"uuid":"0e987a73-8b36-4f26-94fd-f0135d6062c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":118169},{"type":"v1","addr":"172.21.15.86:6817","nonce":118169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":118169},{"type":"v1","addr":"172.21.15.86:6819","nonce":118169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":118169},{"type":"v1","addr":"172.21.15.86:6823","nonce":118169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":118169},{"type":"v1","addr":"172.21.15.86:6821","nonce":118169}]},"public_addr":"172.21.15.86:6817/118169","cluster_addr":"172.21.15.86:6819/118169","heartbeat_back_addr":"172.21.15.86:6823/118169","heartbeat_front_addr":"172.21.15.86:6821/118169","state":["exists","up"]},{"osd":6,"uuid":"7e40bfd5-0f9f-448c-8ccf-2f44bf82a622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":118175},{"type":"v1","addr":"172.21.15.86:6825","nonce":118175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":118175},{"type":"v1","addr":"172.21.15.86:6827","nonce":118175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":118175},{"type":"v1","addr":"172.21.15.86:6831","nonce":118175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":118175},{"type":"v1","addr":"172.21.15.86:6829","nonce":118175}]},"public_addr":"172.21.15.86:6825/118175","cluster_addr":"172.21.15.86:6827/118175","heartbeat_back_addr":"172.21.15.86:6831/118175","heartbeat_front_addr":"172.21.15.86:6829/118175","state":["exists","up"]},{"osd":7,"uuid":"2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":118145},{"type":"v1","addr":"172.21.15.86:6809","nonce":118145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":118145},{"type":"v1","addr":"172.21.15.86:6811","nonce":118145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":118145},{"type":"v1","addr":"172.21.15.86:6815","nonce":118145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":118145},{"type":"v1","addr":"172.21.15.86:6813","nonce":118145}]},"public_addr":"172.21.15.86:6809/118145","cluster_addr":"172.21.15.86:6811/118145","heartbeat_back_addr":"172.21.15.86:6815/118145","heartbeat_front_addr":"172.21.15.86:6813/118145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.319777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.692469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.685180+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.648752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.461468+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.727692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.701726+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.703200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T00:40:21.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 -- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f8210066e90 msgr2=0x7f8210069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f8210066e90 0x7f8210069340 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8218007fd0 tx=0x7f8218009040 comp rx=0 tx=0).stop 2023-06-23T00:40:21.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 -- 172.21.15.64:0/86906432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82300d04f0 msgr2=0x7f8230148d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:21.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82300d04f0 0x7f8230148d50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8220009290 tx=0x7f8220017000 comp rx=0 tx=0).stop 2023-06-23T00:40:21.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 -- 172.21.15.64:0/86906432 shutdown_connections 2023-06-23T00:40:21.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f8210066e90 0x7f8210069340 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82301570c0 0x7f82300c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82300d04f0 0x7f8230148d50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 --2- 172.21.15.64:0/86906432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82300d08c0 0x7f82300b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:21.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 -- 172.21.15.64:0/86906432 >> 172.21.15.64:0/86906432 conn(0x7f823006a060 msgr2=0x7f8230155bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:21.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 -- 172.21.15.64:0/86906432 shutdown_connections 2023-06-23T00:40:21.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:21.934+0000 7f8234b38700 1 -- 172.21.15.64:0/86906432 wait complete. 2023-06-23T00:40:21.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T00:40:22.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.079+0000 7f18e9f6f700 1 Processor -- start 2023-06-23T00:40:22.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.079+0000 7f18e9f6f700 1 -- start start 2023-06-23T00:40:22.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e9f6f700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f18e4151380 0x7f18e4151750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e9f6f700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f18e40cad50 0x7f18e415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e9f6f700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e40cc380 con 0x7f18e41581a0 2023-06-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e9f6f700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e415bbf0 con 0x7f18e40cad50 2023-06-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e9f6f700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e4158640 con 0x7f18e4151380 2023-06-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e3fff700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f18e41581a0 0x7f18e40ca640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43842/0 (socket says 172.21.15.64:43842) 2023-06-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e976e700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f18e40cad50 0x7f18e415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e3fff700 1 -- 172.21.15.64:0/1060082924 learned_addr learned my addr 172.21.15.64:0/1060082924 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e8f6d700 1 --2- 172.21.15.64:0/1060082924 >> v2:172.21.15.86:3300/0 conn(0x7f18e4151380 0x7f18e4151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e976e700 1 -- 172.21.15.64:0/1060082924 >> v2:172.21.15.86:3300/0 conn(0x7f18e4151380 msgr2=0x7f18e4151750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e976e700 1 --2- 172.21.15.64:0/1060082924 >> v2:172.21.15.86:3300/0 conn(0x7f18e4151380 0x7f18e4151750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e976e700 1 -- 172.21.15.64:0/1060082924 >> v1:172.21.15.64:6789/0 conn(0x7f18e41581a0 legacy=0x7f18e40ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e976e700 1 -- 172.21.15.64:0/1060082924 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e415b350 con 0x7f18e40cad50 2023-06-23T00:40:22.086 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.080+0000 7f18e37fe700 1 -- 172.21.15.64:0/1060082924 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803540442 0 0) 0x7f18e40cc380 con 0x7f18e41581a0 2023-06-23T00:40:22.086 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e976e700 1 --2- 172.21.15.64:0/1060082924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18e40cad50 0x7f18e415ac00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f18e406ce90 tx=0x7f18dc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac023a28dc7d828 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:22.086 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e37fe700 1 -- 172.21.15.64:0/1060082924 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc00e070 con 0x7f18e40cad50 2023-06-23T00:40:22.086 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e37fe700 1 -- 172.21.15.64:0/1060082924 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18dc00bb60 con 0x7f18e40cad50 2023-06-23T00:40:22.086 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e37fe700 1 -- 172.21.15.64:0/1060082924 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc012430 con 0x7f18e40cad50 2023-06-23T00:40:22.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e9f6f700 1 -- 172.21.15.64:0/1060082924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18e40cad50 msgr2=0x7f18e415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e9f6f700 1 --2- 172.21.15.64:0/1060082924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18e40cad50 0x7f18e415ac00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f18e406ce90 tx=0x7f18dc009780 comp rx=0 tx=0).stop 2023-06-23T00:40:22.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e9f6f700 1 -- 172.21.15.64:0/1060082924 shutdown_connections 2023-06-23T00:40:22.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e9f6f700 1 --2- 172.21.15.64:0/1060082924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18e40cad50 0x7f18e415ac00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e9f6f700 1 --2- 172.21.15.64:0/1060082924 >> v2:172.21.15.86:3300/0 conn(0x7f18e4151380 0x7f18e4151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e9f6f700 1 -- 172.21.15.64:0/1060082924 >> 172.21.15.64:0/1060082924 conn(0x7f18e406bfc0 msgr2=0x7f18e406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:22.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e9f6f700 1 -- 172.21.15.64:0/1060082924 shutdown_connections 2023-06-23T00:40:22.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.081+0000 7f18e9f6f700 1 -- 172.21.15.64:0/1060082924 wait complete. 2023-06-23T00:40:22.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.082+0000 7f18e9f6f700 1 Processor -- start 2023-06-23T00:40:22.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.083+0000 7f18e9f6f700 1 -- start start 2023-06-23T00:40:22.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.083+0000 7f18e9f6f700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18e40cad50 0x7f18e40c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.083+0000 7f18e9f6f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18e4151380 0x7f18e40c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.083+0000 7f18e9f6f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18e41581a0 0x7f18e4150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.083+0000 7f18e9f6f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40c8880 con 0x7f18e41581a0 2023-06-23T00:40:22.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.083+0000 7f18e9f6f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40b2150 con 0x7f18e4151380 2023-06-23T00:40:22.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.083+0000 7f18e9f6f700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e40c7fe0 con 0x7f18e40cad50 2023-06-23T00:40:22.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.083+0000 7f18e976e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18e41581a0 0x7f18e4150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.083+0000 7f18e976e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18e41581a0 0x7f18e4150590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41088/0 (socket says 172.21.15.64:41088) 2023-06-23T00:40:22.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.083+0000 7f18e976e700 1 -- 172.21.15.64:0/2259989049 learned_addr learned my addr 172.21.15.64:0/2259989049 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:22.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e8f6d700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18e40cad50 0x7f18e40c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e3fff700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18e4151380 0x7f18e40c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e976e700 1 -- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18e40cad50 msgr2=0x7f18e40c7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e976e700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18e40cad50 0x7f18e40c7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e976e700 1 -- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18e4151380 msgr2=0x7f18e40c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e976e700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18e4151380 0x7f18e40c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e976e700 1 -- 172.21.15.64:0/2259989049 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc009a20 con 0x7f18e41581a0 2023-06-23T00:40:22.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e3fff700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18e4151380 0x7f18e40c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T00:40:22.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e976e700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18e41581a0 0x7f18e4150590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009fd0 tx=0x7f18dc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:22.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e0ff9700 1 -- 172.21.15.64:0/2259989049 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc009420 con 0x7f18e41581a0 2023-06-23T00:40:22.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e0ff9700 1 -- 172.21.15.64:0/2259989049 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18dc012e00 con 0x7f18e41581a0 2023-06-23T00:40:22.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e9f6f700 1 -- 172.21.15.64:0/2259989049 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e40c82c0 con 0x7f18e41581a0 2023-06-23T00:40:22.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e0ff9700 1 -- 172.21.15.64:0/2259989049 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc01d860 con 0x7f18e41581a0 2023-06-23T00:40:22.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.084+0000 7f18e9f6f700 1 -- 172.21.15.64:0/2259989049 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e40c7cb0 con 0x7f18e41581a0 2023-06-23T00:40:22.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.085+0000 7f18e9f6f700 1 -- 172.21.15.64:0/2259989049 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e40b2300 con 0x7f18e41581a0 2023-06-23T00:40:22.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.085+0000 7f18e0ff9700 1 -- 172.21.15.64:0/2259989049 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f18dc024070 con 0x7f18e41581a0 2023-06-23T00:40:22.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.086+0000 7f18e0ff9700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f18c4066e90 0x7f18c4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.086+0000 7f18e0ff9700 1 -- 172.21.15.64:0/2259989049 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f18dc087020 con 0x7f18e41581a0 2023-06-23T00:40:22.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.086+0000 7f18e8f6d700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f18c4066e90 0x7f18c4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.086+0000 7f18e8f6d700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f18c4066e90 0x7f18c4069340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009d40 tx=0x7f18cc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:22.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.090+0000 7f18e0ff9700 1 -- 172.21.15.64:0/2259989049 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f18e40b2300 con 0x7f18e41581a0 2023-06-23T00:40:22.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T00:40:22.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T00:40:22.139 INFO:tasks.ceph.mds.d.smithi086.stderr:2023-06-23T00:40:22.136+0000 7ffabeb3c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-06-23T00:40:22.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.335+0000 7f18e9f6f700 1 -- 172.21.15.64:0/2259989049 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18e40b22c0 con 0x7f18e41581a0 2023-06-23T00:40:22.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.335+0000 7f18e0ff9700 1 -- 172.21.15.64:0/2259989049 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4613 (secure 0 0 0) 0x7f18dc053e60 con 0x7f18e41581a0 2023-06-23T00:40:22.338 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:22.339 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6834/1840968204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6833","nonce":1840968204},{"type":"v1","addr":"172.21.15.64:6834","nonce":1840968204}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4279,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/1141461387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1141461387},{"type":"v1","addr":"172.21.15.86:6835","nonce":1141461387}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/1370273364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1370273364},{"type":"v1","addr":"172.21.15.64:6835","nonce":1370273364}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-06-23T00:40:16.414564+0000","modified":"2023-06-23T00:40:20.857359+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/895538016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":895538016},{"type":"v1","addr":"172.21.15.86:6837","nonce":895538016}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-06-23T00:40:22.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 -- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f18c4066e90 msgr2=0x7f18c4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f18c4066e90 0x7f18c4069340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009d40 tx=0x7f18cc008040 comp rx=0 tx=0).stop 2023-06-23T00:40:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 -- 172.21.15.64:0/2259989049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18e41581a0 msgr2=0x7f18e4150590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18e41581a0 0x7f18e4150590 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f18dc009fd0 tx=0x7f18dc019000 comp rx=0 tx=0).stop 2023-06-23T00:40:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 -- 172.21.15.64:0/2259989049 shutdown_connections 2023-06-23T00:40:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f18c4066e90 0x7f18c4069340 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18e41581a0 0x7f18e4150590 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18e40cad50 0x7f18e40c7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 --2- 172.21.15.64:0/2259989049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18e4151380 0x7f18e40c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 -- 172.21.15.64:0/2259989049 >> 172.21.15.64:0/2259989049 conn(0x7f18e406bfc0 msgr2=0x7f18e40c45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 -- 172.21.15.64:0/2259989049 shutdown_connections 2023-06-23T00:40:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.337+0000 7f18e9f6f700 1 -- 172.21.15.64:0/2259989049 wait complete. 2023-06-23T00:40:22.342 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 7 2023-06-23T00:40:22.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-06-23T00:40:16.414564+0000', 'modified': '2023-06-23T00:40:20.857359+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4282': {'gid': 4282, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/895538016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 895538016}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 895538016}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2023-06-23T00:40:22.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-06-23T00:40:22.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-06-23T00:40:22.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.876+0000 7f33ddc8c700 1 Processor -- start 2023-06-23T00:40:22.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.876+0000 7f33ddc8c700 1 -- start start 2023-06-23T00:40:22.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33ddc8c700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f33d815d330 0x7f33d80b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33ddc8c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33d815cbc0 0x7f33d815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33ddc8c700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d815e0e0 con 0x7f33d815d700 2023-06-23T00:40:22.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33ddc8c700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d806a530 con 0x7f33d815d330 2023-06-23T00:40:22.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33ddc8c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33d80b2710 con 0x7f33d815cbc0 2023-06-23T00:40:22.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33dcc8a700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f33d815d330 0x7f33d80b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33d7fff700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f33d815d700 0x7f33d815c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43858/0 (socket says 172.21.15.64:43858) 2023-06-23T00:40:22.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33d7fff700 1 -- 172.21.15.64:0/2841626071 learned_addr learned my addr 172.21.15.64:0/2841626071 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:22.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33dd48b700 1 --2- 172.21.15.64:0/2841626071 >> v2:172.21.15.86:3300/0 conn(0x7f33d815cbc0 0x7f33d815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33dcc8a700 1 -- 172.21.15.64:0/2841626071 >> v2:172.21.15.86:3300/0 conn(0x7f33d815cbc0 msgr2=0x7f33d815dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:22.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/2841626071 >> v2:172.21.15.86:3300/0 conn(0x7f33d815cbc0 0x7f33d815dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33dcc8a700 1 -- 172.21.15.64:0/2841626071 >> v1:172.21.15.64:6789/0 conn(0x7f33d815d700 legacy=0x7f33d815c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33dcc8a700 1 -- 172.21.15.64:0/2841626071 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d806adf0 con 0x7f33d815d330 2023-06-23T00:40:22.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.877+0000 7f33d77fe700 1 -- 172.21.15.64:0/2841626071 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147903253 0 0) 0x7f33d815e0e0 con 0x7f33d815d700 2023-06-23T00:40:22.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/2841626071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33d815d330 0x7f33d80b21d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f33d80cca70 tx=0x7f33d0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a19d4d5db611e87e server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:22.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33d77fe700 1 -- 172.21.15.64:0/2841626071 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d000e070 con 0x7f33d815d330 2023-06-23T00:40:22.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33d77fe700 1 -- 172.21.15.64:0/2841626071 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d000bc30 con 0x7f33d815d330 2023-06-23T00:40:22.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33d77fe700 1 -- 172.21.15.64:0/2841626071 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0012430 con 0x7f33d815d330 2023-06-23T00:40:22.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2841626071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33d815d330 msgr2=0x7f33d80b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33ddc8c700 1 --2- 172.21.15.64:0/2841626071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33d815d330 0x7f33d80b21d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f33d80cca70 tx=0x7f33d0009780 comp rx=0 tx=0).stop 2023-06-23T00:40:22.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2841626071 shutdown_connections 2023-06-23T00:40:22.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33ddc8c700 1 --2- 172.21.15.64:0/2841626071 >> v2:172.21.15.86:3300/0 conn(0x7f33d815cbc0 0x7f33d815dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33ddc8c700 1 --2- 172.21.15.64:0/2841626071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33d815d330 0x7f33d80b21d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2841626071 >> 172.21.15.64:0/2841626071 conn(0x7f33d80d0b30 msgr2=0x7f33d80ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:22.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2841626071 shutdown_connections 2023-06-23T00:40:22.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.878+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2841626071 wait complete. 2023-06-23T00:40:22.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.879+0000 7f33ddc8c700 1 Processor -- start 2023-06-23T00:40:22.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.880+0000 7f33ddc8c700 1 -- start start 2023-06-23T00:40:22.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.880+0000 7f33ddc8c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33d815cbc0 0x7f33d816c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.880+0000 7f33ddc8c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33d815d330 0x7f33d80c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.880+0000 7f33ddc8c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33d815d700 0x7f33d80c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.880+0000 7f33ddc8c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d80c6e50 con 0x7f33d815cbc0 2023-06-23T00:40:22.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.880+0000 7f33ddc8c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d80c6240 con 0x7f33d815d330 2023-06-23T00:40:22.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.880+0000 7f33ddc8c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33d80c6520 con 0x7f33d815d700 2023-06-23T00:40:22.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.880+0000 7f33dcc8a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33d815cbc0 0x7f33d816c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.880+0000 7f33dcc8a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33d815cbc0 0x7f33d816c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41108/0 (socket says 172.21.15.64:41108) 2023-06-23T00:40:22.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33dcc8a700 1 -- 172.21.15.64:0/2914751334 learned_addr learned my addr 172.21.15.64:0/2914751334 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:22.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33d7fff700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33d815d330 0x7f33d80c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33dd48b700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33d815d700 0x7f33d80c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33dcc8a700 1 -- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33d815d700 msgr2=0x7f33d80c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33d815d700 0x7f33d80c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33dcc8a700 1 -- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33d815d330 msgr2=0x7f33d80c58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33d815d330 0x7f33d80c58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33dcc8a700 1 -- 172.21.15.64:0/2914751334 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d0009a20 con 0x7f33d815cbc0 2023-06-23T00:40:22.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33d815cbc0 0x7f33d816c440 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33d0013000 tx=0x7f33d0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:22.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33d4ff9700 1 -- 172.21.15.64:0/2914751334 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0001090 con 0x7f33d815cbc0 2023-06-23T00:40:22.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33d4ff9700 1 -- 172.21.15.64:0/2914751334 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d0027470 con 0x7f33d815cbc0 2023-06-23T00:40:22.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2914751334 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d80c6800 con 0x7f33d815cbc0 2023-06-23T00:40:22.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33d4ff9700 1 -- 172.21.15.64:0/2914751334 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d0027810 con 0x7f33d815cbc0 2023-06-23T00:40:22.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.881+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2914751334 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d816c980 con 0x7f33d815cbc0 2023-06-23T00:40:22.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.882+0000 7f33d4ff9700 1 -- 172.21.15.64:0/2914751334 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f33d0024070 con 0x7f33d815cbc0 2023-06-23T00:40:22.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.883+0000 7f33d4ff9700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c4066e90 0x7f33c4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.883+0000 7f33d4ff9700 1 -- 172.21.15.64:0/2914751334 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f33d0086e40 con 0x7f33d815cbc0 2023-06-23T00:40:22.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.883+0000 7f33d7fff700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c4066e90 0x7f33c4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.883+0000 7f33d7fff700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c4066e90 0x7f33c4069340 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f33d80a6ff0 tx=0x7f33cc007f30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:22.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.885+0000 7f33e81e6b80 1 Processor -- start 2023-06-23T00:40:22.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.885+0000 7f33e81e6b80 1 -- start start 2023-06-23T00:40:22.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.885+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8ba4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.885+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8ba5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.885+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8bcb280 0x5602b8bcb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.885+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602b8ab05f0 con 0x5602b8ba4dc0 2023-06-23T00:40:22.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.885+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602b8af9f10 con 0x5602b8bcb280 2023-06-23T00:40:22.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.885+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5602b8af3780 con 0x5602b8ba69c0 2023-06-23T00:40:22.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33dcc8a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8ba4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33dcc8a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8ba4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41124/0 (socket says 172.21.15.64:41124) 2023-06-23T00:40:22.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33dcc8a700 1 -- 172.21.15.64:0/2351316858 learned_addr learned my addr 172.21.15.64:0/2351316858 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:22.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33d7fff700 1 --2- 172.21.15.64:0/2351316858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8ba5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33dd48b700 1 --2- 172.21.15.64:0/2351316858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8bcb280 0x5602b8bcb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33dcc8a700 1 -- 172.21.15.64:0/2351316858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 msgr2=0x5602b8ba5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:22.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/2351316858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8ba5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33dcc8a700 1 -- 172.21.15.64:0/2351316858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8bcb280 msgr2=0x5602b8bcb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:22.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/2351316858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8bcb280 0x5602b8bcb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33dcc8a700 1 -- 172.21.15.64:0/2351316858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602b898e270 con 0x5602b8ba4dc0 2023-06-23T00:40:22.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/2351316858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8ba4340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f33d0084ee0 tx=0x7f33d0020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb5634f5c40b53f server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:22.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33bdffb700 1 -- 172.21.15.64:0/2351316858 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d00263d0 con 0x5602b8ba4dc0 2023-06-23T00:40:22.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33bdffb700 1 -- 172.21.15.64:0/2351316858 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d0058070 con 0x5602b8ba4dc0 2023-06-23T00:40:22.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.886+0000 7f33bdffb700 1 -- 172.21.15.64:0/2351316858 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d005abc0 con 0x5602b8ba4dc0 2023-06-23T00:40:22.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.887+0000 7f33e81e6b80 1 -- 172.21.15.64:0/2351316858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 msgr2=0x5602b8ba4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.887+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/2351316858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8ba4340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f33d0084ee0 tx=0x7f33d0020040 comp rx=0 tx=0).stop 2023-06-23T00:40:22.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.887+0000 7f33e81e6b80 1 -- 172.21.15.64:0/2351316858 shutdown_connections 2023-06-23T00:40:22.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.887+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/2351316858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8bcb280 0x5602b8bcb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.887+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/2351316858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8ba4340 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.887+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/2351316858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8ba5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.887+0000 7f33e81e6b80 1 -- 172.21.15.64:0/2351316858 >> 172.21.15.64:0/2351316858 conn(0x5602b8ba5870 msgr2=0x5602b8ba5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:22.898 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.887+0000 7f33e81e6b80 1 -- 172.21.15.64:0/2351316858 shutdown_connections 2023-06-23T00:40:22.898 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.887+0000 7f33e81e6b80 1 -- 172.21.15.64:0/2351316858 wait complete. 2023-06-23T00:40:22.901 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.887+0000 7f33e81e6b80 1 -- start start 2023-06-23T00:40:22.901 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33e81e6b80 1 Processor -- start 2023-06-23T00:40:22.901 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba4dc0 0x5602b8bd10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.901 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8bfdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8bcb280 0x5602b8bca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602b8a9f890 con 0x5602b8bcb280 2023-06-23T00:40:22.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602b8abaea0 con 0x5602b8ba4dc0 2023-06-23T00:40:22.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5602b8976e30 con 0x5602b8ba69c0 2023-06-23T00:40:22.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33e81e6b80 10 client.0 _ll_register_callbacks cb 0x5602b8990f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-23T00:40:22.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33dd48b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8bcb280 0x5602b8bca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33dd48b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8bcb280 0x5602b8bca1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41134/0 (socket says 172.21.15.64:41134) 2023-06-23T00:40:22.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33dd48b700 1 -- 172.21.15.64:0/3351132236 learned_addr learned my addr 172.21.15.64:0/3351132236 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:22.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba4dc0 0x5602b8bd10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.888+0000 7f33d7fff700 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8bfdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33dd48b700 1 -- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 msgr2=0x5602b8bfdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:22.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33dd48b700 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8bfdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33dd48b700 1 -- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba4dc0 msgr2=0x5602b8bd10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:22.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33dd48b700 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba4dc0 0x5602b8bd10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33dd48b700 1 -- 172.21.15.64:0/3351132236 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d816c980 con 0x5602b8bcb280 2023-06-23T00:40:22.904 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33d7fff700 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8bfdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T00:40:22.904 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33dd48b700 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8bcb280 0x5602b8bca1c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f33c8013790 tx=0x7f33c8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de659cb14891946c server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:22.904 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33b6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-06-23T00:40:22.904 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33b6ffd700 1 -- 172.21.15.64:0/3351132236 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8015700 con 0x5602b8bcb280 2023-06-23T00:40:22.904 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33b6ffd700 1 -- 172.21.15.64:0/3351132236 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f33c8015860 con 0x5602b8bcb280 2023-06-23T00:40:22.904 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3351132236 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5602b8b98430 con 0x5602b8bcb280 2023-06-23T00:40:22.904 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33b6ffd700 1 -- 172.21.15.64:0/3351132236 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33c8015c30 con 0x5602b8bcb280 2023-06-23T00:40:22.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33e81e6b80 10 client.4434 fetch_fsmap learned FSMap version 7 2023-06-23T00:40:22.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3351132236 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5602b8930fe0 con 0x5602b8bcb280 2023-06-23T00:40:22.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33b6ffd700 1 -- 172.21.15.64:0/3351132236 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5602b8b98430 con 0x5602b8bcb280 2023-06-23T00:40:22.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33b6ffd700 1 -- 172.21.15.64:0/3351132236 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 7) v1 ==== 3116+0+0 (secure 0 0 0) 0x7f33c801b020 con 0x5602b8bcb280 2023-06-23T00:40:22.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33e81e6b80 10 client.4434 fetch_fsmap finished waiting for FSMap version 7 2023-06-23T00:40:22.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33e81e6b80 10 client.4434 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-23T00:40:22.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33e81e6b80 20 client.4434 populate_metadata read hostname 'smithi064' 2023-06-23T00:40:22.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:6836/895538016,v1:172.21.15.86:6837/895538016] conn(0x5602b8bf9bf0 0x5602b8bf9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33e81e6b80 4 client.4434 mds_command: new command op to 4282 tid=0[{"prefix": "get_command_descriptions"}] 2023-06-23T00:40:22.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.889+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3351132236 --> [v2:172.21.15.86:6836/895538016,v1:172.21.15.86:6837/895538016] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5602b8a96870 con 0x5602b8bf9bf0 2023-06-23T00:40:22.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.890+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:6836/895538016,v1:172.21.15.86:6837/895538016] conn(0x5602b8bf9bf0 0x5602b8bf9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.890+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:6836/895538016,v1:172.21.15.86:6837/895538016] conn(0x5602b8bf9bf0 0x5602b8bf9fc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3983b8d107dc74a server_cookie=711b34639dadcb86 in_seq=0 out_seq=0 2023-06-23T00:40:22.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.890+0000 7f33b6ffd700 10 client.4434 ms_handle_connect on v2:172.21.15.86:6836/895538016 2023-06-23T00:40:22.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33b6ffd700 1 -- 172.21.15.64:0/3351132236 <== mds.0 v2:172.21.15.86:6836/895538016 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17215 (crc 0 0 0) 0x5602b8a96870 con 0x5602b8bf9bf0 2023-06-23T00:40:22.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33b6ffd700 10 client.4434 handle_command_reply: tid=0 2023-06-23T00:40:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 1 client.4434 shutdown 2023-06-23T00:40:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3351132236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8bcb280 msgr2=0x5602b8bca1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8bcb280 0x5602b8bca1c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f33c8013790 tx=0x7f33c8008040 comp rx=0 tx=0).stop 2023-06-23T00:40:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3351132236 shutdown_connections 2023-06-23T00:40:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:6836/895538016,v1:172.21.15.86:6837/895538016] conn(0x5602b8bf9bf0 0x5602b8bf9fc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8bcb280 0x5602b8bca1c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33b6ffd700 0 client.4434 ms_handle_reset on v2:172.21.15.86:6836/895538016 2023-06-23T00:40:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba4dc0 0x5602b8bd10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/3351132236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8bfdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3351132236 >> 172.21.15.64:0/3351132236 conn(0x5602b8ba5870 msgr2=0x5602b8be77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3351132236 shutdown_connections 2023-06-23T00:40:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3351132236 wait complete. 2023-06-23T00:40:22.908 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.891+0000 7f33e81e6b80 20 client.4434 trim_cache size 0 max 16384 2023-06-23T00:40:22.910 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.908+0000 7f33e81e6b80 1 Processor -- start 2023-06-23T00:40:22.911 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.908+0000 7f33e81e6b80 1 -- start start 2023-06-23T00:40:22.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba5870 0x5602b8ba0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba4dc0 0x5602b8bd3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba69c0 0x5602b8bc91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602b8bed560 con 0x5602b8ba69c0 2023-06-23T00:40:22.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602b8be89c0 con 0x5602b8ba5870 2023-06-23T00:40:22.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5602b8bd0850 con 0x5602b8ba4dc0 2023-06-23T00:40:22.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33dd48b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba69c0 0x5602b8bc91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33dd48b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba69c0 0x5602b8bc91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41140/0 (socket says 172.21.15.64:41140) 2023-06-23T00:40:22.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33dd48b700 1 -- 172.21.15.64:0/3412960510 learned_addr learned my addr 172.21.15.64:0/3412960510 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:22.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/3412960510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba5870 0x5602b8ba0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33d7fff700 1 --2- 172.21.15.64:0/3412960510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba4dc0 0x5602b8bd3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33dd48b700 1 -- 172.21.15.64:0/3412960510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba4dc0 msgr2=0x5602b8bd3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:22.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33dd48b700 1 --2- 172.21.15.64:0/3412960510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba4dc0 0x5602b8bd3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33dd48b700 1 -- 172.21.15.64:0/3412960510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba5870 msgr2=0x5602b8ba0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:22.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33dd48b700 1 --2- 172.21.15.64:0/3412960510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba5870 0x5602b8ba0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33dd48b700 1 -- 172.21.15.64:0/3412960510 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602b8ac2780 con 0x5602b8ba69c0 2023-06-23T00:40:22.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33dd48b700 1 --2- 172.21.15.64:0/3412960510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba69c0 0x5602b8bc91e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f33c8007430 tx=0x7f33c8023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19cefabf6024d5fb server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:22.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.909+0000 7f33be7fc700 1 -- 172.21.15.64:0/3412960510 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c802e070 con 0x5602b8ba69c0 2023-06-23T00:40:22.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.910+0000 7f33be7fc700 1 -- 172.21.15.64:0/3412960510 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c8013e30 con 0x5602b8ba69c0 2023-06-23T00:40:22.915 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.910+0000 7f33be7fc700 1 -- 172.21.15.64:0/3412960510 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8000be0 con 0x5602b8ba69c0 2023-06-23T00:40:22.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.910+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3412960510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba69c0 msgr2=0x5602b8bc91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.910+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/3412960510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba69c0 0x5602b8bc91e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f33c8007430 tx=0x7f33c8023000 comp rx=0 tx=0).stop 2023-06-23T00:40:22.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.910+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3412960510 shutdown_connections 2023-06-23T00:40:22.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.910+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/3412960510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba69c0 0x5602b8bc91e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.910+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/3412960510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba5870 0x5602b8ba0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.910+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/3412960510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba4dc0 0x5602b8bd3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.910+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3412960510 >> 172.21.15.64:0/3412960510 conn(0x5602b8bfd6e0 msgr2=0x5602b8ba2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:22.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.910+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3412960510 shutdown_connections 2023-06-23T00:40:22.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.910+0000 7f33e81e6b80 1 -- 172.21.15.64:0/3412960510 wait complete. 2023-06-23T00:40:22.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.911+0000 7f33e81e6b80 1 -- start start 2023-06-23T00:40:22.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.911+0000 7f33e81e6b80 1 Processor -- start 2023-06-23T00:40:22.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.911+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8c37b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.911+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba5870 0x5602b8bee820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.911+0000 7f33e81e6b80 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8bcc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.911+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602b8b99c20 con 0x5602b8ba4dc0 2023-06-23T00:40:22.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.911+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602b8bcc550 con 0x5602b8ba5870 2023-06-23T00:40:22.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.911+0000 7f33e81e6b80 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5602b8b99730 con 0x5602b8ba69c0 2023-06-23T00:40:22.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33e81e6b80 10 client.0 _ll_register_callbacks cb 0x5602b8990f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-06-23T00:40:22.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33dcc8a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8c37b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33dcc8a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8c37b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41154/0 (socket says 172.21.15.64:41154) 2023-06-23T00:40:22.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33dcc8a700 1 -- 172.21.15.64:0/566598518 learned_addr learned my addr 172.21.15.64:0/566598518 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:22.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33dd48b700 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8bcc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33d7fff700 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba5870 0x5602b8bee820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33dcc8a700 1 -- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 msgr2=0x5602b8bcc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:22.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8bcc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33dcc8a700 1 -- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba5870 msgr2=0x5602b8bee820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:22.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba5870 0x5602b8bee820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33dcc8a700 1 -- 172.21.15.64:0/566598518 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602b8930fe0 con 0x5602b8ba4dc0 2023-06-23T00:40:22.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33dcc8a700 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8c37b80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009fd0 tx=0x7f33d0020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706223bf34aacdbb server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:22.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33b6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-06-23T00:40:22.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33b6ffd700 1 -- 172.21.15.64:0/566598518 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d005e740 con 0x5602b8ba4dc0 2023-06-23T00:40:22.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33b6ffd700 1 -- 172.21.15.64:0/566598518 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f33d005abc0 con 0x5602b8ba4dc0 2023-06-23T00:40:22.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33e81e6b80 1 -- 172.21.15.64:0/566598518 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33d0058070 con 0x5602b8ba4dc0 2023-06-23T00:40:22.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33b6ffd700 1 -- 172.21.15.64:0/566598518 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33d0057340 con 0x5602b8ba4dc0 2023-06-23T00:40:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33e81e6b80 10 client.4440 fetch_fsmap learned FSMap version 7 2023-06-23T00:40:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33e81e6b80 1 -- 172.21.15.64:0/566598518 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5602b8aa9470 con 0x5602b8ba4dc0 2023-06-23T00:40:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.912+0000 7f33b6ffd700 1 -- 172.21.15.64:0/566598518 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d0058070 con 0x5602b8ba4dc0 2023-06-23T00:40:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33b6ffd700 1 -- 172.21.15.64:0/566598518 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 7) v1 ==== 3116+0+0 (secure 0 0 0) 0x7f33d0056ed0 con 0x5602b8ba4dc0 2023-06-23T00:40:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33e81e6b80 10 client.4440 fetch_fsmap finished waiting for FSMap version 7 2023-06-23T00:40:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33e81e6b80 10 client.4440 resolve_mds: resolved name 'd' to daemon mds.d 2023-06-23T00:40:22.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33e81e6b80 20 client.4440 populate_metadata read hostname 'smithi064' 2023-06-23T00:40:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:6836/895538016,v1:172.21.15.86:6837/895538016] conn(0x5602b8bcb280 0x5602b8c51e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33e81e6b80 4 client.4440 mds_command: new command op to 4282 tid=0[{"prefix": "status"}] 2023-06-23T00:40:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33e81e6b80 1 -- 172.21.15.64:0/566598518 --> [v2:172.21.15.86:6836/895538016,v1:172.21.15.86:6837/895538016] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5602b8a96590 con 0x5602b8bcb280 2023-06-23T00:40:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33d7fff700 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:6836/895538016,v1:172.21.15.86:6837/895538016] conn(0x5602b8bcb280 0x5602b8c51e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33b6ffd700 10 client.4440 ms_handle_connect on v2:172.21.15.86:6836/895538016 2023-06-23T00:40:22.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33d7fff700 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:6836/895538016,v1:172.21.15.86:6837/895538016] conn(0x5602b8bcb280 0x5602b8c51e60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b765ad338b0834b2 server_cookie=b1a4a75f1b40c1de in_seq=0 out_seq=0 2023-06-23T00:40:22.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33b6ffd700 1 -- 172.21.15.64:0/566598518 <== mds.0 v2:172.21.15.86:6836/895538016 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x5602b8a96590 con 0x5602b8bcb280 2023-06-23T00:40:22.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33b6ffd700 10 client.4440 handle_command_reply: tid=0 2023-06-23T00:40:22.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.913+0000 7f33e81e6b80 1 client.4440 shutdown 2023-06-23T00:40:22.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33e81e6b80 1 -- 172.21.15.64:0/566598518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 msgr2=0x5602b8c37b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8c37b80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f33d0009fd0 tx=0x7f33d0020040 comp rx=0 tx=0).stop 2023-06-23T00:40:22.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33e81e6b80 1 -- 172.21.15.64:0/566598518 shutdown_connections 2023-06-23T00:40:22.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:6836/895538016,v1:172.21.15.86:6837/895538016] conn(0x5602b8bcb280 0x5602b8c51e60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5602b8ba69c0 0x5602b8bcc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33b6ffd700 0 client.4440 ms_handle_reset on v2:172.21.15.86:6836/895538016 2023-06-23T00:40:22.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602b8ba4dc0 0x5602b8c37b80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33e81e6b80 1 --2- 172.21.15.64:0/566598518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5602b8ba5870 0x5602b8bee820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33e81e6b80 1 -- 172.21.15.64:0/566598518 >> 172.21.15.64:0/566598518 conn(0x5602b8bfd6e0 msgr2=0x5602b8ba0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:22.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33e81e6b80 1 -- 172.21.15.64:0/566598518 shutdown_connections 2023-06-23T00:40:22.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33e81e6b80 1 -- 172.21.15.64:0/566598518 wait complete. 2023-06-23T00:40:22.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33e81e6b80 20 client.4440 trim_cache size 0 max 16384 2023-06-23T00:40:22.929 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-06-23T00:40:22.930 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "72efde62-4c91-4277-87ee-356e50bd4cf5", 2023-06-23T00:40:22.930 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 0, 2023-06-23T00:40:22.930 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 4282, 2023-06-23T00:40:22.930 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-06-23T00:40:22.930 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-06-23T00:40:22.930 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-06-23T00:40:22.930 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 6.4974663250000004, 2023-06-23T00:40:22.931 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 7, 2023-06-23T00:40:22.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 24, 2023-06-23T00:40:22.931 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 23, 2023-06-23T00:40:22.931 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 11.827101884999999 2023-06-23T00:40:22.931 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-06-23T00:40:22.932 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c4066e90 msgr2=0x7f33c4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.933 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c4066e90 0x7f33c4069340 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f33d80a6ff0 tx=0x7f33cc007f30 comp rx=0 tx=0).stop 2023-06-23T00:40:22.933 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2914751334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33d815cbc0 msgr2=0x7f33d816c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:22.933 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33d815cbc0 0x7f33d816c440 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33d0013000 tx=0x7f33d0019000 comp rx=0 tx=0).stop 2023-06-23T00:40:22.933 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2914751334 shutdown_connections 2023-06-23T00:40:22.933 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f33c4066e90 0x7f33c4069340 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.933 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33d815d700 0x7f33d80c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.934 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33d815cbc0 0x7f33d816c440 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.934 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 --2- 172.21.15.64:0/2914751334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33d815d330 0x7f33d80c58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:22.934 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2914751334 >> 172.21.15.64:0/2914751334 conn(0x7f33d80d0b30 msgr2=0x7f33d80afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:22.934 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2914751334 shutdown_connections 2023-06-23T00:40:22.934 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:22.914+0000 7f33ddc8c700 1 -- 172.21.15.64:0/2914751334 wait complete. 2023-06-23T00:40:22.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-06-23T00:40:23.054 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.051+0000 7f86033af700 1 Processor -- start 2023-06-23T00:40:23.054 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.051+0000 7f86033af700 1 -- start start 2023-06-23T00:40:23.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.052+0000 7f86033af700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f85fc1570c0 0x7f85fc15a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:23.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.052+0000 7f86033af700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85fc15aaa0 con 0x7f85fc0d04f0 2023-06-23T00:40:23.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.052+0000 7f86033af700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85fc0ce5c0 con 0x7f85fc0d08c0 2023-06-23T00:40:23.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.052+0000 7f86033af700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85fc0cd1d0 con 0x7f85fc1570c0 2023-06-23T00:40:23.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.052+0000 7f860114b700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f85fc0d04f0 0x7f85fc1569b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43860/0 (socket says 172.21.15.64:43860) 2023-06-23T00:40:23.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.052+0000 7f860114b700 1 -- 172.21.15.64:0/488278447 learned_addr learned my addr 172.21.15.64:0/488278447 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:23.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.053+0000 7f860094a700 1 --2- 172.21.15.64:0/488278447 >> v2:172.21.15.86:3300/0 conn(0x7f85fc1570c0 0x7f85fc15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:23.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.053+0000 7f85fbfff700 1 -- 172.21.15.64:0/488278447 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750533290 0 0) 0x7f85fc15aaa0 con 0x7f85fc0d04f0 2023-06-23T00:40:23.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.053+0000 7f85fbfff700 1 -- 172.21.15.64:0/488278447 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e4003570 con 0x7f85fc0d04f0 2023-06-23T00:40:23.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.053+0000 7f85fbfff700 1 -- 172.21.15.64:0/488278447 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 266523145 0 0) 0x7f85e4003570 con 0x7f85fc0d04f0 2023-06-23T00:40:23.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.053+0000 7f860094a700 1 -- 172.21.15.64:0/488278447 >> v1:172.21.15.86:6789/0 conn(0x7f85fc0d08c0 legacy=0x7f85fc0cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:23.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.053+0000 7f860094a700 1 -- 172.21.15.64:0/488278447 >> v1:172.21.15.64:6789/0 conn(0x7f85fc0d04f0 legacy=0x7f85fc1569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:23.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.053+0000 7f860094a700 1 -- 172.21.15.64:0/488278447 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85fc0cd4b0 con 0x7f85fc1570c0 2023-06-23T00:40:23.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.053+0000 7f860094a700 1 --2- 172.21.15.64:0/488278447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85fc1570c0 0x7f85fc15a410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85f00098e0 tx=0x7f85f00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dafed7b5b60fedbc server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:23.060 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.053+0000 7f85fbfff700 1 -- 172.21.15.64:0/488278447 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f40017f0 con 0x7f85fc1570c0 2023-06-23T00:40:23.060 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.053+0000 7f85fbfff700 1 -- 172.21.15.64:0/488278447 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f0011420 con 0x7f85fc1570c0 2023-06-23T00:40:23.060 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.053+0000 7f85fbfff700 1 -- 172.21.15.64:0/488278447 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f00117a0 con 0x7f85fc1570c0 2023-06-23T00:40:23.060 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.054+0000 7f86033af700 1 -- 172.21.15.64:0/488278447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85fc1570c0 msgr2=0x7f85fc15a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:23.060 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.054+0000 7f86033af700 1 --2- 172.21.15.64:0/488278447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85fc1570c0 0x7f85fc15a410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85f00098e0 tx=0x7f85f00091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:23.060 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.054+0000 7f86033af700 1 -- 172.21.15.64:0/488278447 shutdown_connections 2023-06-23T00:40:23.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.054+0000 7f86033af700 1 --2- 172.21.15.64:0/488278447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85fc1570c0 0x7f85fc15a410 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:23.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.054+0000 7f86033af700 1 -- 172.21.15.64:0/488278447 >> 172.21.15.64:0/488278447 conn(0x7f85fc06a060 msgr2=0x7f85fc06a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:23.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.054+0000 7f86033af700 1 -- 172.21.15.64:0/488278447 shutdown_connections 2023-06-23T00:40:23.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.054+0000 7f86033af700 1 -- 172.21.15.64:0/488278447 wait complete. 2023-06-23T00:40:23.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.055+0000 7f86033af700 1 Processor -- start 2023-06-23T00:40:23.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.055+0000 7f86033af700 1 -- start start 2023-06-23T00:40:23.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f86033af700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85fc0d04f0 0x7f85fc0875f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:23.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f86033af700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85fc0d08c0 0x7f85fc14dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:23.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f86033af700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85fc1570c0 0x7f85fc0c5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:23.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f86033af700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc14e140 con 0x7f85fc0d04f0 2023-06-23T00:40:23.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f86033af700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc1591a0 con 0x7f85fc0d08c0 2023-06-23T00:40:23.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f86033af700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85fc087c80 con 0x7f85fc1570c0 2023-06-23T00:40:23.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860114b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85fc0d04f0 0x7f85fc0875f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:23.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860114b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85fc0d04f0 0x7f85fc0875f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41166/0 (socket says 172.21.15.64:41166) 2023-06-23T00:40:23.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860114b700 1 -- 172.21.15.64:0/3196097438 learned_addr learned my addr 172.21.15.64:0/3196097438 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:23.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860094a700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85fc0d08c0 0x7f85fc14dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:23.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860194c700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85fc1570c0 0x7f85fc0c5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:23.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860114b700 1 -- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85fc1570c0 msgr2=0x7f85fc0c5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:23.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860114b700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85fc1570c0 0x7f85fc0c5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:23.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860114b700 1 -- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85fc0d08c0 msgr2=0x7f85fc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:23.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860114b700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85fc0d08c0 0x7f85fc14dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:23.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860114b700 1 -- 172.21.15.64:0/3196097438 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f0009420 con 0x7f85fc0d04f0 2023-06-23T00:40:23.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860094a700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85fc0d08c0 0x7f85fc14dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T00:40:23.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f860114b700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85fc0d04f0 0x7f85fc0875f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f85f400cc10 tx=0x7f85f4001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:23.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.056+0000 7f85f97fa700 1 -- 172.21.15.64:0/3196097438 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4004dc0 con 0x7f85fc0d04f0 2023-06-23T00:40:23.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.057+0000 7f85f97fa700 1 -- 172.21.15.64:0/3196097438 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f400a6b0 con 0x7f85fc0d04f0 2023-06-23T00:40:23.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.057+0000 7f86033af700 1 -- 172.21.15.64:0/3196097438 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85fc0c50e0 con 0x7f85fc0d04f0 2023-06-23T00:40:23.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.057+0000 7f85f97fa700 1 -- 172.21.15.64:0/3196097438 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f400aa70 con 0x7f85fc0d04f0 2023-06-23T00:40:23.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.057+0000 7f86033af700 1 -- 172.21.15.64:0/3196097438 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85fc0c56a0 con 0x7f85fc0d04f0 2023-06-23T00:40:23.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.057+0000 7f86033af700 1 -- 172.21.15.64:0/3196097438 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85fc04c1b0 con 0x7f85fc0d04f0 2023-06-23T00:40:23.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.058+0000 7f85f97fa700 1 -- 172.21.15.64:0/3196097438 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f85f4007150 con 0x7f85fc0d04f0 2023-06-23T00:40:23.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.058+0000 7f85f97fa700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f85e8066e90 0x7f85e8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:23.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.058+0000 7f85f97fa700 1 -- 172.21.15.64:0/3196097438 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f85f4084370 con 0x7f85fc0d04f0 2023-06-23T00:40:23.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.058+0000 7f860094a700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f85e8066e90 0x7f85e8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:23.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.059+0000 7f860094a700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f85e8066e90 0x7f85e8069340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85f00097b0 tx=0x7f85f0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:23.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.062+0000 7f85f97fa700 1 -- 172.21.15.64:0/3196097438 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f85f40512d0 con 0x7f85fc0d04f0 2023-06-23T00:40:23.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.305+0000 7f86033af700 1 -- 172.21.15.64:0/3196097438 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f85fc0c6160 con 0x7f85fc0d04f0 2023-06-23T00:40:23.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.305+0000 7f85f97fa700 1 -- 172.21.15.64:0/3196097438 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f85f4018070 con 0x7f85fc0d04f0 2023-06-23T00:40:23.308 INFO:teuthology.orchestra.run.smithi064.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-06-23T00:40:23.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 -- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f85e8066e90 msgr2=0x7f85e8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:23.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f85e8066e90 0x7f85e8069340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85f00097b0 tx=0x7f85f0016000 comp rx=0 tx=0).stop 2023-06-23T00:40:23.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 -- 172.21.15.64:0/3196097438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85fc0d04f0 msgr2=0x7f85fc0875f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:23.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85fc0d04f0 0x7f85fc0875f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f85f400cc10 tx=0x7f85f4001640 comp rx=0 tx=0).stop 2023-06-23T00:40:23.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 -- 172.21.15.64:0/3196097438 shutdown_connections 2023-06-23T00:40:23.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f85e8066e90 0x7f85e8069340 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:23.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85fc1570c0 0x7f85fc0c5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:23.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85fc0d04f0 0x7f85fc0875f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:23.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 --2- 172.21.15.64:0/3196097438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85fc0d08c0 0x7f85fc14dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:23.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 -- 172.21.15.64:0/3196097438 >> 172.21.15.64:0/3196097438 conn(0x7f85fc06a060 msgr2=0x7f85fc0ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:23.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 -- 172.21.15.64:0/3196097438 shutdown_connections 2023-06-23T00:40:23.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:23.307+0000 7f86033af700 1 -- 172.21.15.64:0/3196097438 wait complete. 2023-06-23T00:40:23.321 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-06-23T00:40:23.322 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-06-23T00:40:23.322 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi064.front.sepia.ceph.com 2023-06-23T00:40:23.322 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-06-23T00:40:23.323 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-06-23T00:40:23.323 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-06-23T00:40:23.323 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-06-23T00:40:23.324 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-06-23T00:40:23.324 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-06-23T00:40:23.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-23T00:40:23.324 DEBUG:teuthology.orchestra.run.smithi064:> ip addr 2023-06-23T00:40:23.339 INFO:teuthology.orchestra.run.smithi064.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-06-23T00:40:23.340 INFO:teuthology.orchestra.run.smithi064.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-06-23T00:40:23.340 INFO:teuthology.orchestra.run.smithi064.stdout: inet 127.0.0.1/8 scope host lo 2023-06-23T00:40:23.340 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2023-06-23T00:40:23.340 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 ::1/128 scope host 2023-06-23T00:40:23.340 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2023-06-23T00:40:23.340 INFO:teuthology.orchestra.run.smithi064.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-23T00:40:23.341 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:78 brd ff:ff:ff:ff:ff:ff 2023-06-23T00:40:23.341 INFO:teuthology.orchestra.run.smithi064.stdout: altname enp7s0f0 2023-06-23T00:40:23.341 INFO:teuthology.orchestra.run.smithi064.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-23T00:40:23.341 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:79 brd ff:ff:ff:ff:ff:ff 2023-06-23T00:40:23.341 INFO:teuthology.orchestra.run.smithi064.stdout: altname enp7s0f1 2023-06-23T00:40:23.341 INFO:teuthology.orchestra.run.smithi064.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-06-23T00:40:23.341 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:de brd ff:ff:ff:ff:ff:ff 2023-06-23T00:40:23.342 INFO:teuthology.orchestra.run.smithi064.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-06-23T00:40:23.342 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:df brd ff:ff:ff:ff:ff:ff 2023-06-23T00:40:23.342 INFO:teuthology.orchestra.run.smithi064.stdout: inet 172.21.15.64/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-06-23T00:40:23.342 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft 41980sec preferred_lft 41980sec 2023-06-23T00:40:23.342 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 fe80::ec4:7aff:feb6:f8df/64 scope link 2023-06-23T00:40:23.342 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2023-06-23T00:40:23.343 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-06-23T00:40:23.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-23T00:40:23.344 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-06-23T00:40:23.344 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-06-23T00:40:23.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add name ceph-brx type bridge 2023-06-23T00:40:23.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr flush dev ceph-brx 2023-06-23T00:40:23.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set ceph-brx up 2023-06-23T00:40:23.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-06-23T00:40:23.346 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-06-23T00:40:23.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-23T00:40:23.526 DEBUG:teuthology.orchestra.run.smithi064:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-06-23T00:40:23.553 INFO:teuthology.orchestra.run.smithi064.stdout:1 2023-06-23T00:40:23.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-23T00:40:23.554 DEBUG:teuthology.orchestra.run.smithi064:> route 2023-06-23T00:40:23.635 INFO:teuthology.orchestra.run.smithi064.stdout:Kernel IP routing table 2023-06-23T00:40:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-06-23T00:40:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-06-23T00:40:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-06-23T00:40:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-06-23T00:40:23.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-23T00:40:23.638 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-06-23T00:40:23.638 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-06-23T00:40:23.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-06-23T00:40:23.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-06-23T00:40:23.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-06-23T00:40:23.640 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-06-23T00:40:23.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-23T00:40:23.888 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2023-06-23T00:40:23.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-23T00:40:23.903 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list-id 2023-06-23T00:40:23.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-23T00:40:23.959 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-06-23T00:40:23.959 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-06-23T00:40:23.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-06-23T00:40:23.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-06-23T00:40:23.960 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-06-23T00:40:24.050 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-06-23T00:40:24.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-23T00:40:24.051 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-06-23T00:40:24.051 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-06-23T00:40:24.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-06-23T00:40:24.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-06-23T00:40:24.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-06-23T00:40:24.052 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-06-23T00:40:24.053 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-06-23T00:40:24.053 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-06-23T00:40:24.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-23T00:40:24.228 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-06-23T00:40:24.228 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-06-23T00:40:24.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set brx.0 up 2023-06-23T00:40:24.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set dev brx.0 master ceph-brx 2023-06-23T00:40:24.229 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-06-23T00:40:24.323 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-06-23T00:40:24.323 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-23T00:40:24.323 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-06-23T00:40:24.338 INFO:teuthology.orchestra.run.smithi064.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-06-23T00:40:24.338 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-06-23T00:40:24.339 DEBUG:teuthology.orchestra.run.smithi064:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-06-23T00:40:24.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo modprobe fuse 2023-06-23T00:40:24.460 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-23T00:40:24.461 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-23T00:40:24.528 INFO:teuthology.orchestra.run.smithi064.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-23T00:40:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-23T00:40:24.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-06-23T00:40:24.530 DEBUG:teuthology.orchestra.run.smithi064:> ls /sys/fs/fuse/connections 2023-06-23T00:40:24.585 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-06-23T00:40:24.585 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-06-23T00:40:24.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo modprobe fuse 2023-06-23T00:40:24.652 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-06-23T00:40:24.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-06-23T00:40:24.718 INFO:teuthology.orchestra.run.smithi064.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-06-23T00:40:24.719 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-06-23T00:40:24.720 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-06-23T00:40:24.721 DEBUG:teuthology.orchestra.run.smithi064:> ls /sys/fs/fuse/connections 2023-06-23T00:40:24.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:2023-06-23T00:40:24.727+0000 7fb209dda540 -1 init, newargv = 0x55a40354e330 newargc=15 2023-06-23T00:40:24.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:ceph-fuse[95648]: starting ceph client 2023-06-23T00:40:24.735 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:ceph-fuse[95648]: starting fuse 2023-06-23T00:40:24.744 INFO:teuthology.orchestra.run.smithi064.stdout:47 2023-06-23T00:40:24.745 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-06-23T00:40:24.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-06-23T00:40:24.745 DEBUG:teuthology.orchestra.run.smithi064:> import glob 2023-06-23T00:40:24.746 DEBUG:teuthology.orchestra.run.smithi064:> import re 2023-06-23T00:40:24.746 DEBUG:teuthology.orchestra.run.smithi064:> import os 2023-06-23T00:40:24.746 DEBUG:teuthology.orchestra.run.smithi064:> import subprocess 2023-06-23T00:40:24.747 DEBUG:teuthology.orchestra.run.smithi064:> 2023-06-23T00:40:24.747 DEBUG:teuthology.orchestra.run.smithi064:> def _find_admin_socket(client_name): 2023-06-23T00:40:24.747 DEBUG:teuthology.orchestra.run.smithi064:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-06-23T00:40:24.747 DEBUG:teuthology.orchestra.run.smithi064:> files = glob.glob(asok_path) 2023-06-23T00:40:24.748 DEBUG:teuthology.orchestra.run.smithi064:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-06-23T00:40:24.748 DEBUG:teuthology.orchestra.run.smithi064:> 2023-06-23T00:40:24.748 DEBUG:teuthology.orchestra.run.smithi064:> # Given a non-glob path, it better be there 2023-06-23T00:40:24.749 DEBUG:teuthology.orchestra.run.smithi064:> if "*" not in asok_path: 2023-06-23T00:40:24.749 DEBUG:teuthology.orchestra.run.smithi064:> assert(len(files) == 1) 2023-06-23T00:40:24.749 DEBUG:teuthology.orchestra.run.smithi064:> return files[0] 2023-06-23T00:40:24.749 DEBUG:teuthology.orchestra.run.smithi064:> 2023-06-23T00:40:24.750 DEBUG:teuthology.orchestra.run.smithi064:> for f in files: 2023-06-23T00:40:24.750 DEBUG:teuthology.orchestra.run.smithi064:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-06-23T00:40:24.750 DEBUG:teuthology.orchestra.run.smithi064:> if os.path.exists("/proc/{0}".format(pid)): 2023-06-23T00:40:24.751 DEBUG:teuthology.orchestra.run.smithi064:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-06-23T00:40:24.751 DEBUG:teuthology.orchestra.run.smithi064:> contents = proc_f.read() 2023-06-23T00:40:24.751 DEBUG:teuthology.orchestra.run.smithi064:> if mountpoint in contents: 2023-06-23T00:40:24.752 DEBUG:teuthology.orchestra.run.smithi064:> return f 2023-06-23T00:40:24.752 DEBUG:teuthology.orchestra.run.smithi064:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-06-23T00:40:24.752 DEBUG:teuthology.orchestra.run.smithi064:> 2023-06-23T00:40:24.753 DEBUG:teuthology.orchestra.run.smithi064:> print(_find_admin_socket("client.admin")) 2023-06-23T00:40:24.753 DEBUG:teuthology.orchestra.run.smithi064:> ' 2023-06-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi064.stdout:/var/run/ceph/ceph-client.admin.95648.asok 2023-06-23T00:40:25.049 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.95648.asok 2023-06-23T00:40:25.049 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-06-23T00:40:25.050 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.95648.asok status 2023-06-23T00:40:25.205 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-06-23T00:40:25.206 INFO:teuthology.orchestra.run.smithi064.stdout: "metadata": { 2023-06-23T00:40:25.206 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_sha1": "42ddc46e6e6582d8e29a5a982a81886fe028be2c", 2023-06-23T00:40:25.206 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable)", 2023-06-23T00:40:25.206 INFO:teuthology.orchestra.run.smithi064.stdout: "entity_id": "admin", 2023-06-23T00:40:25.206 INFO:teuthology.orchestra.run.smithi064.stdout: "hostname": "smithi064", 2023-06-23T00:40:25.206 INFO:teuthology.orchestra.run.smithi064.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-06-23T00:40:25.206 INFO:teuthology.orchestra.run.smithi064.stdout: "pid": "95648", 2023-06-23T00:40:25.207 INFO:teuthology.orchestra.run.smithi064.stdout: "root": "/" 2023-06-23T00:40:25.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-06-23T00:40:25.207 INFO:teuthology.orchestra.run.smithi064.stdout: "dentry_count": 0, 2023-06-23T00:40:25.207 INFO:teuthology.orchestra.run.smithi064.stdout: "dentry_pinned_count": 0, 2023-06-23T00:40:25.207 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 4455, 2023-06-23T00:40:25.207 INFO:teuthology.orchestra.run.smithi064.stdout: "inst": { 2023-06-23T00:40:25.207 INFO:teuthology.orchestra.run.smithi064.stdout: "name": { 2023-06-23T00:40:25.207 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "client", 2023-06-23T00:40:25.208 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 4455 2023-06-23T00:40:25.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-06-23T00:40:25.208 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": { 2023-06-23T00:40:25.208 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2023-06-23T00:40:25.208 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "192.168.0.1:0", 2023-06-23T00:40:25.208 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 173749183 2023-06-23T00:40:25.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-06-23T00:40:25.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-06-23T00:40:25.209 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": { 2023-06-23T00:40:25.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2023-06-23T00:40:25.209 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "192.168.0.1:0", 2023-06-23T00:40:25.209 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 173749183 2023-06-23T00:40:25.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-06-23T00:40:25.209 INFO:teuthology.orchestra.run.smithi064.stdout: "inst_str": "client.4455 192.168.0.1:0/173749183", 2023-06-23T00:40:25.209 INFO:teuthology.orchestra.run.smithi064.stdout: "addr_str": "192.168.0.1:0/173749183", 2023-06-23T00:40:25.210 INFO:teuthology.orchestra.run.smithi064.stdout: "inode_count": 1, 2023-06-23T00:40:25.210 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_epoch": 7, 2023-06-23T00:40:25.210 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_epoch": 24, 2023-06-23T00:40:25.210 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_epoch_barrier": 0, 2023-06-23T00:40:25.210 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklisted": false, 2023-06-23T00:40:25.210 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs" 2023-06-23T00:40:25.210 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-06-23T00:40:25.215 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-06-23T00:40:25.216 DEBUG:teuthology.orchestra.run.smithi064:> stat --file-system '--printf=%T 2023-06-23T00:40:25.216 DEBUG:teuthology.orchestra.run.smithi064:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-06-23T00:40:25.231 INFO:teuthology.orchestra.run.smithi064.stdout:fuseblk 2023-06-23T00:40:25.231 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-06-23T00:40:25.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-06-23T00:40:25.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-06-23T00:40:25.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-06-23T00:40:25.298 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-06-23T00:40:25.391 INFO:teuthology.orchestra.run.smithi064.stdout:# file: . 2023-06-23T00:40:25.392 INFO:teuthology.orchestra.run.smithi064.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-06-23T00:40:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:25.392 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi064.front.sepia.ceph.com... 2023-06-23T00:40:25.392 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-06-23T00:40:25.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-06-23T00:40:25.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:ceph-fuse[95648]: fuse finished with error 0 and tester_r 0 2023-06-23T00:40:25.426 INFO:teuthology.orchestra.run:waiting for 900 2023-06-23T00:40:25.521 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:40:25.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-06-23T00:40:25.521 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-06-23T00:40:25.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T00:40:25.697 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.693+0000 7f401938c700 1 Processor -- start 2023-06-23T00:40:25.697 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.693+0000 7f401938c700 1 -- start start 2023-06-23T00:40:25.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.694+0000 7f401938c700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4014153b10 0x7f40140cd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:25.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.694+0000 7f401938c700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401415b7b0 con 0x7f4014153740 2023-06-23T00:40:25.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.694+0000 7f401938c700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f401415aba0 con 0x7f401415cf30 2023-06-23T00:40:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.694+0000 7f401938c700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40140cba90 con 0x7f4014153b10 2023-06-23T00:40:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.694+0000 7f4012ffd700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f4014153740 0x7f40140cb380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43866/0 (socket says 172.21.15.64:43866) 2023-06-23T00:40:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.694+0000 7f4012ffd700 1 -- 172.21.15.64:0/18365803 learned_addr learned my addr 172.21.15.64:0/18365803 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:25.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f40127fc700 1 --2- 172.21.15.64:0/18365803 >> v2:172.21.15.86:3300/0 conn(0x7f4014153b10 0x7f40140cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926419135 0 0) 0x7f401415b7b0 con 0x7f4014153740 2023-06-23T00:40:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4000003570 con 0x7f4014153740 2023-06-23T00:40:25.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4150887118 0 0) 0x7f4000003570 con 0x7f4014153740 2023-06-23T00:40:25.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 >> v2:172.21.15.86:3300/0 conn(0x7f4014153b10 msgr2=0x7f40140cd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:25.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 --2- 172.21.15.64:0/18365803 >> v2:172.21.15.86:3300/0 conn(0x7f4014153b10 0x7f40140cd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:25.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 >> v1:172.21.15.86:6789/0 conn(0x7f401415cf30 legacy=0x7f401415bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:25.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401415ae80 con 0x7f4014153740 2023-06-23T00:40:25.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 939283155 0 0) 0x7f400c001470 con 0x7f4014153740 2023-06-23T00:40:25.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f40127fc700 1 --2- 172.21.15.64:0/18365803 >> v2:172.21.15.86:3300/0 conn(0x7f4014153b10 0x7f40140cd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:25.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 >> v1:172.21.15.64:6789/0 conn(0x7f4014153740 legacy=0x7f40140cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:25.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 --2- 172.21.15.64:0/18365803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 0x7f4000003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:25.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 --2- 172.21.15.64:0/18365803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4000004810 0x7f4000004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:25.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 --2- 172.21.15.64:0/18365803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000051c0 0x7f400000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:25.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400000c130 con 0x7f4000003a60 2023-06-23T00:40:25.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400000c2a0 con 0x7f40000051c0 2023-06-23T00:40:25.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400000c460 con 0x7f4000004810 2023-06-23T00:40:25.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4012ffd700 1 --2- 172.21.15.64:0/18365803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 0x7f4000003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:25.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f40127fc700 1 --2- 172.21.15.64:0/18365803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4000004810 0x7f4000004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f40137fe700 1 --2- 172.21.15.64:0/18365803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000051c0 0x7f400000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4012ffd700 1 -- 172.21.15.64:0/18365803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4000004810 msgr2=0x7f4000004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4012ffd700 1 --2- 172.21.15.64:0/18365803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4000004810 0x7f4000004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4012ffd700 1 -- 172.21.15.64:0/18365803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000051c0 msgr2=0x7f400000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4012ffd700 1 --2- 172.21.15.64:0/18365803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40000051c0 0x7f400000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f40137fe700 1 -- 172.21.15.64:0/18365803 reap_dead start 2023-06-23T00:40:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.695+0000 7f4012ffd700 1 -- 172.21.15.64:0/18365803 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4000003570 con 0x7f4000003a60 2023-06-23T00:40:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.696+0000 7f4012ffd700 1 --2- 172.21.15.64:0/18365803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 0x7f4000003e70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f400c00c3c0 tx=0x7f400c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3111622814418d31 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.696+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00cc90 con 0x7f4000003a60 2023-06-23T00:40:25.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.696+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c00cdf0 con 0x7f4000003a60 2023-06-23T00:40:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.696+0000 7f4011ffb700 1 -- 172.21.15.64:0/18365803 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c005b50 con 0x7f4000003a60 2023-06-23T00:40:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.696+0000 7f401938c700 1 -- 172.21.15.64:0/18365803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 msgr2=0x7f4000003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.696+0000 7f401938c700 1 --2- 172.21.15.64:0/18365803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 0x7f4000003e70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f400c00c3c0 tx=0x7f400c001e50 comp rx=0 tx=0).stop 2023-06-23T00:40:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.696+0000 7f401938c700 1 -- 172.21.15.64:0/18365803 shutdown_connections 2023-06-23T00:40:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.696+0000 7f401938c700 1 --2- 172.21.15.64:0/18365803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 0x7f4000003e70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.696+0000 7f401938c700 1 -- 172.21.15.64:0/18365803 >> 172.21.15.64:0/18365803 conn(0x7f401406aaa0 msgr2=0x7f40140871f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.696+0000 7f401938c700 1 -- 172.21.15.64:0/18365803 shutdown_connections 2023-06-23T00:40:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.696+0000 7f401938c700 1 -- 172.21.15.64:0/18365803 wait complete. 2023-06-23T00:40:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.697+0000 7f401938c700 1 Processor -- start 2023-06-23T00:40:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f401938c700 1 -- start start 2023-06-23T00:40:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f401938c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 0x7f40140b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f401938c700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4014150b70 0x7f4014152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f401938c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4014158d40 0x7f40141599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f401938c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40141588b0 con 0x7f4000003a60 2023-06-23T00:40:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f401938c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40140b5260 con 0x7f4014158d40 2023-06-23T00:40:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f401938c700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40140b5540 con 0x7f4014150b70 2023-06-23T00:40:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f4012ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 0x7f40140b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f4012ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 0x7f40140b4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41182/0 (socket says 172.21.15.64:41182) 2023-06-23T00:40:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f4012ffd700 1 -- 172.21.15.64:0/2012476270 learned_addr learned my addr 172.21.15.64:0/2012476270 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f40137fe700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4014158d40 0x7f40141599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:25.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.698+0000 7f40127fc700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4014150b70 0x7f4014152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:25.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f4012ffd700 1 -- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4014150b70 msgr2=0x7f4014152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f4012ffd700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4014150b70 0x7f4014152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f4012ffd700 1 -- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4014158d40 msgr2=0x7f40141599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f4012ffd700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4014158d40 0x7f40141599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f4012ffd700 1 -- 172.21.15.64:0/2012476270 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40140b5820 con 0x7f4000003a60 2023-06-23T00:40:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f40137fe700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4014158d40 0x7f40141599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T00:40:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f4012ffd700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 0x7f40140b4c80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f400000c270 tx=0x7f400c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f400affd700 1 -- 172.21.15.64:0/2012476270 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c00cc90 con 0x7f4000003a60 2023-06-23T00:40:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f400affd700 1 -- 172.21.15.64:0/2012476270 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c0070d0 con 0x7f4000003a60 2023-06-23T00:40:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f401938c700 1 -- 172.21.15.64:0/2012476270 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4014158540 con 0x7f4000003a60 2023-06-23T00:40:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f400affd700 1 -- 172.21.15.64:0/2012476270 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c007490 con 0x7f4000003a60 2023-06-23T00:40:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.699+0000 7f401938c700 1 -- 172.21.15.64:0/2012476270 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4014150fe0 con 0x7f4000003a60 2023-06-23T00:40:25.721 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.700+0000 7f401938c700 1 -- 172.21.15.64:0/2012476270 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4014158b90 con 0x7f4000003a60 2023-06-23T00:40:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.701+0000 7f400affd700 1 -- 172.21.15.64:0/2012476270 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f400c028460 con 0x7f4000003a60 2023-06-23T00:40:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.701+0000 7f400affd700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ff4066e90 0x7f3ff4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.701+0000 7f400affd700 1 -- 172.21.15.64:0/2012476270 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f400c085d90 con 0x7f4000003a60 2023-06-23T00:40:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.701+0000 7f40127fc700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ff4066e90 0x7f3ff4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:25.722 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.702+0000 7f40127fc700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ff4066e90 0x7f3ff4069340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f40140c9620 tx=0x7f3ffc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:25.723 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.705+0000 7f400affd700 1 -- 172.21.15.64:0/2012476270 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f400c052c70 con 0x7f4000003a60 2023-06-23T00:40:25.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.900+0000 7f401938c700 1 -- 172.21.15.64:0/2012476270 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f401404c1b0 con 0x7f4000003a60 2023-06-23T00:40:25.910 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.901+0000 7f400affd700 1 -- 172.21.15.64:0/2012476270 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f400c017090 con 0x7f4000003a60 2023-06-23T00:40:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:25.911 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":24,"fsid":"72efde62-4c91-4277-87ee-356e50bd4cf5","created":"2023-06-23T00:39:58.078232+0000","modified":"2023-06-23T00:40:20.857067+0000","last_up_change":"2023-06-23T00:40:08.853479+0000","last_in_change":"2023-06-23T00:40:00.493656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-23T00:40:09.294447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-23T00:40:09.719568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T00:40:14.182408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T00:40:15.395198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-23T00:40:18.457653+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":88027},{"type":"v1","addr":"172.21.15.64:6801","nonce":88027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":88027},{"type":"v1","addr":"172.21.15.64:6803","nonce":88027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":88027},{"type":"v1","addr":"172.21.15.64:6807","nonce":88027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":88027},{"type":"v1","addr":"172.21.15.64:6805","nonce":88027}]},"public_addr":"172.21.15.64:6801/88027","cluster_addr":"172.21.15.64:6803/88027","heartbeat_back_addr":"172.21.15.64:6807/88027","heartbeat_front_addr":"172.21.15.64:6805/88027","state":["exists","up"]},{"osd":1,"uuid":"f0698c9d-c67a-42ac-8d36-838836c514d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":88062},{"type":"v1","addr":"172.21.15.64:6825","nonce":88062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":88062},{"type":"v1","addr":"172.21.15.64:6827","nonce":88062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":88062},{"type":"v1","addr":"172.21.15.64:6831","nonce":88062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":88062},{"type":"v1","addr":"172.21.15.64:6829","nonce":88062}]},"public_addr":"172.21.15.64:6825/88062","cluster_addr":"172.21.15.64:6827/88062","heartbeat_back_addr":"172.21.15.64:6831/88062","heartbeat_front_addr":"172.21.15.64:6829/88062","state":["exists","up"]},{"osd":2,"uuid":"a5ece869-9da7-40e8-8f1d-bbaf67bb2187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":88045},{"type":"v1","addr":"172.21.15.64:6817","nonce":88045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":88045},{"type":"v1","addr":"172.21.15.64:6819","nonce":88045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":88045},{"type":"v1","addr":"172.21.15.64:6823","nonce":88045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":88045},{"type":"v1","addr":"172.21.15.64:6821","nonce":88045}]},"public_addr":"172.21.15.64:6817/88045","cluster_addr":"172.21.15.64:6819/88045","heartbeat_back_addr":"172.21.15.64:6823/88045","heartbeat_front_addr":"172.21.15.64:6821/88045","state":["exists","up"]},{"osd":3,"uuid":"21067fac-663b-4ecf-ae67-bf235161638e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":88044},{"type":"v1","addr":"172.21.15.64:6809","nonce":88044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":88044},{"type":"v1","addr":"172.21.15.64:6811","nonce":88044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":88044},{"type":"v1","addr":"172.21.15.64:6815","nonce":88044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":88044},{"type":"v1","addr":"172.21.15.64:6813","nonce":88044}]},"public_addr":"172.21.15.64:6809/88044","cluster_addr":"172.21.15.64:6811/88044","heartbeat_back_addr":"172.21.15.64:6815/88044","heartbeat_front_addr":"172.21.15.64:6813/88044","state":["exists","up"]},{"osd":4,"uuid":"6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":118144},{"type":"v1","addr":"172.21.15.86:6801","nonce":118144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":118144},{"type":"v1","addr":"172.21.15.86:6803","nonce":118144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":118144},{"type":"v1","addr":"172.21.15.86:6807","nonce":118144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":118144},{"type":"v1","addr":"172.21.15.86:6805","nonce":118144}]},"public_addr":"172.21.15.86:6801/118144","cluster_addr":"172.21.15.86:6803/118144","heartbeat_back_addr":"172.21.15.86:6807/118144","heartbeat_front_addr":"172.21.15.86:6805/118144","state":["exists","up"]},{"osd":5,"uuid":"0e987a73-8b36-4f26-94fd-f0135d6062c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":118169},{"type":"v1","addr":"172.21.15.86:6817","nonce":118169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":118169},{"type":"v1","addr":"172.21.15.86:6819","nonce":118169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":118169},{"type":"v1","addr":"172.21.15.86:6823","nonce":118169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":118169},{"type":"v1","addr":"172.21.15.86:6821","nonce":118169}]},"public_addr":"172.21.15.86:6817/118169","cluster_addr":"172.21.15.86:6819/118169","heartbeat_back_addr":"172.21.15.86:6823/118169","heartbeat_front_addr":"172.21.15.86:6821/118169","state":["exists","up"]},{"osd":6,"uuid":"7e40bfd5-0f9f-448c-8ccf-2f44bf82a622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":118175},{"type":"v1","addr":"172.21.15.86:6825","nonce":118175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":118175},{"type":"v1","addr":"172.21.15.86:6827","nonce":118175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":118175},{"type":"v1","addr":"172.21.15.86:6831","nonce":118175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":118175},{"type":"v1","addr":"172.21.15.86:6829","nonce":118175}]},"public_addr":"172.21.15.86:6825/118175","cluster_addr":"172.21.15.86:6827/118175","heartbeat_back_addr":"172.21.15.86:6831/118175","heartbeat_front_addr":"172.21.15.86:6829/118175","state":["exists","up"]},{"osd":7,"uuid":"2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":118145},{"type":"v1","addr":"172.21.15.86:6809","nonce":118145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":118145},{"type":"v1","addr":"172.21.15.86:6811","nonce":118145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":118145},{"type":"v1","addr":"172.21.15.86:6815","nonce":118145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":118145},{"type":"v1","addr":"172.21.15.86:6813","nonce":118145}]},"public_addr":"172.21.15.86:6809/118145","cluster_addr":"172.21.15.86:6811/118145","heartbeat_back_addr":"172.21.15.86:6815/118145","heartbeat_front_addr":"172.21.15.86:6813/118145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.319777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.692469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.685180+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.648752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.461468+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.727692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.701726+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.703200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T00:40:25.912 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 -- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ff4066e90 msgr2=0x7f3ff4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:25.912 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ff4066e90 0x7f3ff4069340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f40140c9620 tx=0x7f3ffc0091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:25.912 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 -- 172.21.15.64:0/2012476270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 msgr2=0x7f40140b4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:25.912 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 0x7f40140b4c80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f400000c270 tx=0x7f400c015000 comp rx=0 tx=0).stop 2023-06-23T00:40:25.912 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 -- 172.21.15.64:0/2012476270 shutdown_connections 2023-06-23T00:40:25.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3ff4066e90 0x7f3ff4069340 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:25.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4014158d40 0x7f40141599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:25.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4000003a60 0x7f40140b4c80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:25.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 --2- 172.21.15.64:0/2012476270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4014150b70 0x7f4014152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:25.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 -- 172.21.15.64:0/2012476270 >> 172.21.15.64:0/2012476270 conn(0x7f401406aaa0 msgr2=0x7f4014151e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:25.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 -- 172.21.15.64:0/2012476270 shutdown_connections 2023-06-23T00:40:25.913 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:25.903+0000 7f401938c700 1 -- 172.21.15.64:0/2012476270 wait complete. 2023-06-23T00:40:25.918 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T00:40:26.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.034+0000 7f29a4845700 1 Processor -- start 2023-06-23T00:40:26.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.034+0000 7f29a4845700 1 -- start start 2023-06-23T00:40:26.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f29a4845700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f29a015d700 0x7f29a015dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f29a4845700 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f29a015c4b0 0x7f29a015c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f29a4845700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a015e0e0 con 0x7f29a015d330 2023-06-23T00:40:26.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f29a4845700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a006a530 con 0x7f29a015d700 2023-06-23T00:40:26.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f29a4845700 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a006adf0 con 0x7f29a015c4b0 2023-06-23T00:40:26.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299e59c700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7f29a015d700 0x7f29a015dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299ed9d700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f29a015d330 0x7f29a00b21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43870/0 (socket says 172.21.15.64:43870) 2023-06-23T00:40:26.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299ed9d700 1 -- 172.21.15.64:0/1929500885 learned_addr learned my addr 172.21.15.64:0/1929500885 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:26.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299f59e700 1 --2- 172.21.15.64:0/1929500885 >> v2:172.21.15.86:3300/0 conn(0x7f29a015c4b0 0x7f29a015c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299dd9b700 1 -- 172.21.15.64:0/1929500885 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994784832 0 0) 0x7f29a015e0e0 con 0x7f29a015d330 2023-06-23T00:40:26.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299dd9b700 1 -- 172.21.15.64:0/1929500885 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2988003570 con 0x7f29a015d330 2023-06-23T00:40:26.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299e59c700 1 -- 172.21.15.64:0/1929500885 >> v2:172.21.15.86:3300/0 conn(0x7f29a015c4b0 msgr2=0x7f29a015c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:26.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299e59c700 1 --2- 172.21.15.64:0/1929500885 >> v2:172.21.15.86:3300/0 conn(0x7f29a015c4b0 0x7f29a015c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299e59c700 1 -- 172.21.15.64:0/1929500885 >> v1:172.21.15.64:6789/0 conn(0x7f29a015d330 legacy=0x7f29a00b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299e59c700 1 -- 172.21.15.64:0/1929500885 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a006a7a0 con 0x7f29a015d700 2023-06-23T00:40:26.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299e59c700 1 --2- 172.21.15.64:0/1929500885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29a015d700 0x7f29a015dba0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29a00cca70 tx=0x7f29940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267dcd68807a847d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:26.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299dd9b700 1 -- 172.21.15.64:0/1929500885 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994011630 con 0x7f29a015d700 2023-06-23T00:40:26.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299dd9b700 1 -- 172.21.15.64:0/1929500885 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29940117b0 con 0x7f29a015d700 2023-06-23T00:40:26.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.035+0000 7f299dd9b700 1 -- 172.21.15.64:0/1929500885 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994011b70 con 0x7f29a015d700 2023-06-23T00:40:26.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.036+0000 7f29a4845700 1 -- 172.21.15.64:0/1929500885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29a015d700 msgr2=0x7f29a015dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.036+0000 7f29a4845700 1 --2- 172.21.15.64:0/1929500885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29a015d700 0x7f29a015dba0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29a00cca70 tx=0x7f29940091a0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.036+0000 7f29a4845700 1 -- 172.21.15.64:0/1929500885 shutdown_connections 2023-06-23T00:40:26.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.036+0000 7f29a4845700 1 --2- 172.21.15.64:0/1929500885 >> v2:172.21.15.86:3300/0 conn(0x7f29a015c4b0 0x7f29a015c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.036+0000 7f29a4845700 1 --2- 172.21.15.64:0/1929500885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29a015d700 0x7f29a015dba0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.036+0000 7f29a4845700 1 -- 172.21.15.64:0/1929500885 >> 172.21.15.64:0/1929500885 conn(0x7f29a00d0b30 msgr2=0x7f29a00ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:26.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.036+0000 7f29a4845700 1 -- 172.21.15.64:0/1929500885 shutdown_connections 2023-06-23T00:40:26.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.036+0000 7f29a4845700 1 -- 172.21.15.64:0/1929500885 wait complete. 2023-06-23T00:40:26.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.037+0000 7f29a4845700 1 Processor -- start 2023-06-23T00:40:26.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.037+0000 7f29a4845700 1 -- start start 2023-06-23T00:40:26.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f29a4845700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a015c4b0 0x7f29a00c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f29a4845700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a015d330 0x7f29a016a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f29a4845700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29a015d700 0x7f29a00c4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f29a4845700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a00a6b10 con 0x7f29a015d700 2023-06-23T00:40:26.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f29a4845700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a016dce0 con 0x7f29a015d330 2023-06-23T00:40:26.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f29a4845700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a016a6e0 con 0x7f29a015c4b0 2023-06-23T00:40:26.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299f59e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29a015d700 0x7f29a00c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299f59e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29a015d700 0x7f29a00c4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41194/0 (socket says 172.21.15.64:41194) 2023-06-23T00:40:26.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299f59e700 1 -- 172.21.15.64:0/1233446353 learned_addr learned my addr 172.21.15.64:0/1233446353 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:26.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299e59c700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a015d330 0x7f29a016a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299ed9d700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a015c4b0 0x7f29a00c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299f59e700 1 -- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a015c4b0 msgr2=0x7f29a00c3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299f59e700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a015c4b0 0x7f29a00c3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299f59e700 1 -- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a015d330 msgr2=0x7f29a016a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299f59e700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a015d330 0x7f29a016a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299f59e700 1 -- 172.21.15.64:0/1233446353 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2994009420 con 0x7f29a015d700 2023-06-23T00:40:26.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299e59c700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a015d330 0x7f29a016a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T00:40:26.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.038+0000 7f299ed9d700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a015c4b0 0x7f29a00c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T00:40:26.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.039+0000 7f299f59e700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29a015d700 0x7f29a00c4540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2998009e90 tx=0x7f299800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:26.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.039+0000 7f2986ffd700 1 -- 172.21.15.64:0/1233446353 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299800cc80 con 0x7f29a015d700 2023-06-23T00:40:26.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.039+0000 7f2986ffd700 1 -- 172.21.15.64:0/1233446353 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299800f040 con 0x7f29a015d700 2023-06-23T00:40:26.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.039+0000 7f29a4845700 1 -- 172.21.15.64:0/1233446353 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a016d0d0 con 0x7f29a015d700 2023-06-23T00:40:26.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.039+0000 7f2986ffd700 1 -- 172.21.15.64:0/1233446353 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2998014480 con 0x7f29a015d700 2023-06-23T00:40:26.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.039+0000 7f29a4845700 1 -- 172.21.15.64:0/1233446353 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a016d690 con 0x7f29a015d700 2023-06-23T00:40:26.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.040+0000 7f29a4845700 1 -- 172.21.15.64:0/1233446353 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a00c8be0 con 0x7f29a015d700 2023-06-23T00:40:26.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.040+0000 7f2986ffd700 1 -- 172.21.15.64:0/1233446353 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f2998014830 con 0x7f29a015d700 2023-06-23T00:40:26.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.040+0000 7f2986ffd700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f298c066e90 0x7f298c069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.041+0000 7f2986ffd700 1 -- 172.21.15.64:0/1233446353 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f29980853c0 con 0x7f29a015d700 2023-06-23T00:40:26.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.041+0000 7f299ed9d700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f298c066e90 0x7f298c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.041+0000 7f299ed9d700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f298c066e90 0x7f298c069340 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f29900058d0 tx=0x7f2990009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:26.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.045+0000 7f2986ffd700 1 -- 172.21.15.64:0/1233446353 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f29980522a0 con 0x7f29a015d700 2023-06-23T00:40:26.245 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.241+0000 7f29a4845700 1 -- 172.21.15.64:0/1233446353 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29a006a280 con 0x7f29a015d700 2023-06-23T00:40:26.245 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.242+0000 7f2986ffd700 1 -- 172.21.15.64:0/1233446353 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f29980558c0 con 0x7f29a015d700 2023-06-23T00:40:26.246 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-06-23T00:40:26.246 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":24,"fsid":"72efde62-4c91-4277-87ee-356e50bd4cf5","created":"2023-06-23T00:39:58.078232+0000","modified":"2023-06-23T00:40:20.857067+0000","last_up_change":"2023-06-23T00:40:08.853479+0000","last_in_change":"2023-06-23T00:40:00.493656+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-23T00:40:09.294447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-23T00:40:09.719568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T00:40:14.182408+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T00:40:15.395198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-06-23T00:40:18.457653+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6ec9702d-d617-4cb1-8aa6-ef1d6f9c1d37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":88027},{"type":"v1","addr":"172.21.15.64:6801","nonce":88027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":88027},{"type":"v1","addr":"172.21.15.64:6803","nonce":88027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":88027},{"type":"v1","addr":"172.21.15.64:6807","nonce":88027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":88027},{"type":"v1","addr":"172.21.15.64:6805","nonce":88027}]},"public_addr":"172.21.15.64:6801/88027","cluster_addr":"172.21.15.64:6803/88027","heartbeat_back_addr":"172.21.15.64:6807/88027","heartbeat_front_addr":"172.21.15.64:6805/88027","state":["exists","up"]},{"osd":1,"uuid":"f0698c9d-c67a-42ac-8d36-838836c514d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":88062},{"type":"v1","addr":"172.21.15.64:6825","nonce":88062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":88062},{"type":"v1","addr":"172.21.15.64:6827","nonce":88062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":88062},{"type":"v1","addr":"172.21.15.64:6831","nonce":88062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":88062},{"type":"v1","addr":"172.21.15.64:6829","nonce":88062}]},"public_addr":"172.21.15.64:6825/88062","cluster_addr":"172.21.15.64:6827/88062","heartbeat_back_addr":"172.21.15.64:6831/88062","heartbeat_front_addr":"172.21.15.64:6829/88062","state":["exists","up"]},{"osd":2,"uuid":"a5ece869-9da7-40e8-8f1d-bbaf67bb2187","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":88045},{"type":"v1","addr":"172.21.15.64:6817","nonce":88045}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":88045},{"type":"v1","addr":"172.21.15.64:6819","nonce":88045}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":88045},{"type":"v1","addr":"172.21.15.64:6823","nonce":88045}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":88045},{"type":"v1","addr":"172.21.15.64:6821","nonce":88045}]},"public_addr":"172.21.15.64:6817/88045","cluster_addr":"172.21.15.64:6819/88045","heartbeat_back_addr":"172.21.15.64:6823/88045","heartbeat_front_addr":"172.21.15.64:6821/88045","state":["exists","up"]},{"osd":3,"uuid":"21067fac-663b-4ecf-ae67-bf235161638e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":88044},{"type":"v1","addr":"172.21.15.64:6809","nonce":88044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":88044},{"type":"v1","addr":"172.21.15.64:6811","nonce":88044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":88044},{"type":"v1","addr":"172.21.15.64:6815","nonce":88044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":88044},{"type":"v1","addr":"172.21.15.64:6813","nonce":88044}]},"public_addr":"172.21.15.64:6809/88044","cluster_addr":"172.21.15.64:6811/88044","heartbeat_back_addr":"172.21.15.64:6815/88044","heartbeat_front_addr":"172.21.15.64:6813/88044","state":["exists","up"]},{"osd":4,"uuid":"6e0c4552-8d5e-4659-8d65-e57e5cfdd2cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":118144},{"type":"v1","addr":"172.21.15.86:6801","nonce":118144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":118144},{"type":"v1","addr":"172.21.15.86:6803","nonce":118144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":118144},{"type":"v1","addr":"172.21.15.86:6807","nonce":118144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":118144},{"type":"v1","addr":"172.21.15.86:6805","nonce":118144}]},"public_addr":"172.21.15.86:6801/118144","cluster_addr":"172.21.15.86:6803/118144","heartbeat_back_addr":"172.21.15.86:6807/118144","heartbeat_front_addr":"172.21.15.86:6805/118144","state":["exists","up"]},{"osd":5,"uuid":"0e987a73-8b36-4f26-94fd-f0135d6062c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":118169},{"type":"v1","addr":"172.21.15.86:6817","nonce":118169}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":118169},{"type":"v1","addr":"172.21.15.86:6819","nonce":118169}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":118169},{"type":"v1","addr":"172.21.15.86:6823","nonce":118169}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":118169},{"type":"v1","addr":"172.21.15.86:6821","nonce":118169}]},"public_addr":"172.21.15.86:6817/118169","cluster_addr":"172.21.15.86:6819/118169","heartbeat_back_addr":"172.21.15.86:6823/118169","heartbeat_front_addr":"172.21.15.86:6821/118169","state":["exists","up"]},{"osd":6,"uuid":"7e40bfd5-0f9f-448c-8ccf-2f44bf82a622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":118175},{"type":"v1","addr":"172.21.15.86:6825","nonce":118175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":118175},{"type":"v1","addr":"172.21.15.86:6827","nonce":118175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":118175},{"type":"v1","addr":"172.21.15.86:6831","nonce":118175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":118175},{"type":"v1","addr":"172.21.15.86:6829","nonce":118175}]},"public_addr":"172.21.15.86:6825/118175","cluster_addr":"172.21.15.86:6827/118175","heartbeat_back_addr":"172.21.15.86:6831/118175","heartbeat_front_addr":"172.21.15.86:6829/118175","state":["exists","up"]},{"osd":7,"uuid":"2e8b1ee6-a743-4826-aa57-3a9fe1f15e2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":118145},{"type":"v1","addr":"172.21.15.86:6809","nonce":118145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":118145},{"type":"v1","addr":"172.21.15.86:6811","nonce":118145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":118145},{"type":"v1","addr":"172.21.15.86:6815","nonce":118145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":118145},{"type":"v1","addr":"172.21.15.86:6813","nonce":118145}]},"public_addr":"172.21.15.86:6809/118145","cluster_addr":"172.21.15.86:6811/118145","heartbeat_back_addr":"172.21.15.86:6815/118145","heartbeat_front_addr":"172.21.15.86:6813/118145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.319777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.692469+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.685180+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.648752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.461468+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.727692+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.701726+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T00:40:05.703200+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T00:40:26.247 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 -- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f298c066e90 msgr2=0x7f298c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f298c066e90 0x7f298c069340 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f29900058d0 tx=0x7f2990009040 comp rx=0 tx=0).stop 2023-06-23T00:40:26.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 -- 172.21.15.64:0/1233446353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29a015d700 msgr2=0x7f29a00c4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29a015d700 0x7f29a00c4540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2998009e90 tx=0x7f299800b040 comp rx=0 tx=0).stop 2023-06-23T00:40:26.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 -- 172.21.15.64:0/1233446353 shutdown_connections 2023-06-23T00:40:26.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f298c066e90 0x7f298c069340 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29a015d700 0x7f29a00c4540 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29a015c4b0 0x7f29a00c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 --2- 172.21.15.64:0/1233446353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29a015d330 0x7f29a016a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 -- 172.21.15.64:0/1233446353 >> 172.21.15.64:0/1233446353 conn(0x7f29a00d0b30 msgr2=0x7f29a00afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:26.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 -- 172.21.15.64:0/1233446353 shutdown_connections 2023-06-23T00:40:26.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.244+0000 7f29a4845700 1 -- 172.21.15.64:0/1233446353 wait complete. 2023-06-23T00:40:26.260 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-23T00:40:26.382 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.378+0000 7f3f1d654700 1 Processor -- start 2023-06-23T00:40:26.382 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.378+0000 7f3f1d654700 1 -- start start 2023-06-23T00:40:26.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.379+0000 7f3f1d654700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f180cb910 0x7f3f1815cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.379+0000 7f3f1d654700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1806a530 con 0x7f3f1815d700 2023-06-23T00:40:26.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.379+0000 7f3f1d654700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1806adf0 con 0x7f3f1815d330 2023-06-23T00:40:26.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.379+0000 7f3f1d654700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1806a7a0 con 0x7f3f180cb910 2023-06-23T00:40:26.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.379+0000 7f3f177fe700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7f3f1815d700 0x7f3f1815c4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43878/0 (socket says 172.21.15.64:43878) 2023-06-23T00:40:26.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.379+0000 7f3f177fe700 1 -- 172.21.15.64:0/3100178716 learned_addr learned my addr 172.21.15.64:0/3100178716 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:26.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.379+0000 7f3f1ce53700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f180cb910 0x7f3f1815cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.379+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582846124 0 0) 0x7f3f1806a530 con 0x7f3f1815d700 2023-06-23T00:40:26.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.379+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 --> v1:172.21.15.64:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f00003570 con 0x7f3f1815d700 2023-06-23T00:40:26.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159040455 0 0) 0x7f3f1806adf0 con 0x7f3f1815d330 2023-06-23T00:40:26.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1806a530 con 0x7f3f1815d330 2023-06-23T00:40:26.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 <== mon.0 v1:172.21.15.64:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2719378029 0 0) 0x7f3f00003570 con 0x7f3f1815d700 2023-06-23T00:40:26.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f180cb910 msgr2=0x7f3f1815cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:26.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f180cb910 0x7f3f1815cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 >> v1:172.21.15.86:6789/0 conn(0x7f3f1815d330 legacy=0x7f3f180b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 --> v1:172.21.15.64:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1806aa80 con 0x7f3f1815d700 2023-06-23T00:40:26.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 <== mon.0 v1:172.21.15.64:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 939283155 0 0) 0x7f3f0c001470 con 0x7f3f1815d700 2023-06-23T00:40:26.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 >> v1:172.21.15.64:6789/0 conn(0x7f3f1815d700 legacy=0x7f3f1815c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f180cb910 0x7f3f00003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.388 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f00004180 0x7f3f00004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.388 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f00004d50 0x7f3f000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.388 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1806a7a0 con 0x7f3f00004180 2023-06-23T00:40:26.388 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0000bba0 con 0x7f3f00004d50 2023-06-23T00:40:26.388 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0000bd60 con 0x7f3f180cb910 2023-06-23T00:40:26.388 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f177fe700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f00004180 0x7f3f00004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f17fff700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f180cb910 0x7f3f00003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f1ce53700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f00004d50 0x7f3f000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f177fe700 1 -- 172.21.15.64:0/3100178716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f180cb910 msgr2=0x7f3f00003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:26.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f177fe700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f180cb910 0x7f3f00003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f177fe700 1 -- 172.21.15.64:0/3100178716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f00004d50 msgr2=0x7f3f000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:26.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f177fe700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f00004d50 0x7f3f000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f177fe700 1 -- 172.21.15.64:0/3100178716 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f00003570 con 0x7f3f00004180 2023-06-23T00:40:26.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f177fe700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f00004180 0x7f3f00004810 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00c2d0 tx=0x7f3f0c001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c18107d6f9821b server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:26.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.380+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c00cb60 con 0x7f3f00004180 2023-06-23T00:40:26.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.381+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c00cce0 con 0x7f3f00004180 2023-06-23T00:40:26.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.381+0000 7f3f16ffd700 1 -- 172.21.15.64:0/3100178716 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c005b70 con 0x7f3f00004180 2023-06-23T00:40:26.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.381+0000 7f3f1d654700 1 -- 172.21.15.64:0/3100178716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f00004180 msgr2=0x7f3f00004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.381+0000 7f3f1d654700 1 --2- 172.21.15.64:0/3100178716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f00004180 0x7f3f00004810 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00c2d0 tx=0x7f3f0c001e10 comp rx=0 tx=0).stop 2023-06-23T00:40:26.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.381+0000 7f3f1ce53700 1 -- 172.21.15.64:0/3100178716 reap_dead start 2023-06-23T00:40:26.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.381+0000 7f3f1d654700 1 -- 172.21.15.64:0/3100178716 shutdown_connections 2023-06-23T00:40:26.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.381+0000 7f3f1d654700 1 -- 172.21.15.64:0/3100178716 >> 172.21.15.64:0/3100178716 conn(0x7f3f180d0b30 msgr2=0x7f3f180ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:26.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.381+0000 7f3f1d654700 1 -- 172.21.15.64:0/3100178716 shutdown_connections 2023-06-23T00:40:26.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.381+0000 7f3f1d654700 1 -- 172.21.15.64:0/3100178716 wait complete. 2023-06-23T00:40:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.382+0000 7f3f1d654700 1 Processor -- start 2023-06-23T00:40:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.382+0000 7f3f1d654700 1 -- start start 2023-06-23T00:40:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f1d654700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f00004180 0x7f3f180c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f1d654700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f180c70d0 0x7f3f1814a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f1d654700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f1814af40 0x7f3f1815fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f1d654700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180c78b0 con 0x7f3f180c70d0 2023-06-23T00:40:26.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f1d654700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180c5280 con 0x7f3f1814af40 2023-06-23T00:40:26.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f1d654700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180c5560 con 0x7f3f00004180 2023-06-23T00:40:26.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f177fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f180c70d0 0x7f3f1814a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f177fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f180c70d0 0x7f3f1814a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41204/0 (socket says 172.21.15.64:41204) 2023-06-23T00:40:26.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f177fe700 1 -- 172.21.15.64:0/3217146429 learned_addr learned my addr 172.21.15.64:0/3217146429 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:26.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f1ce53700 1 --2- 172.21.15.64:0/3217146429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f1814af40 0x7f3f1815fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f177fe700 1 -- 172.21.15.64:0/3217146429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f00004180 msgr2=0x7f3f180c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.383+0000 7f3f17fff700 1 --2- 172.21.15.64:0/3217146429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f00004180 0x7f3f180c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3f177fe700 1 --2- 172.21.15.64:0/3217146429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f00004180 0x7f3f180c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3f177fe700 1 -- 172.21.15.64:0/3217146429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f1814af40 msgr2=0x7f3f1815fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3f177fe700 1 --2- 172.21.15.64:0/3217146429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f1814af40 0x7f3f1815fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3f177fe700 1 -- 172.21.15.64:0/3217146429 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180c5840 con 0x7f3f180c70d0 2023-06-23T00:40:26.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3f17fff700 1 --2- 172.21.15.64:0/3217146429 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f00004180 0x7f3f180c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T00:40:26.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3f1ce53700 1 --2- 172.21.15.64:0/3217146429 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f1814af40 0x7f3f1815fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T00:40:26.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3f177fe700 1 --2- 172.21.15.64:0/3217146429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f180c70d0 0x7f3f1814a680 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c001120 tx=0x7f3f0c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:26.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3efffff700 1 -- 172.21.15.64:0/3217146429 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c00ce50 con 0x7f3f180c70d0 2023-06-23T00:40:26.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3efffff700 1 -- 172.21.15.64:0/3217146429 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c0072e0 con 0x7f3f180c70d0 2023-06-23T00:40:26.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3efffff700 1 -- 172.21.15.64:0/3217146429 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c0076e0 con 0x7f3f180c70d0 2023-06-23T00:40:26.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3f1d654700 1 -- 172.21.15.64:0/3217146429 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f180c5b20 con 0x7f3f180c70d0 2023-06-23T00:40:26.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.384+0000 7f3f1d654700 1 -- 172.21.15.64:0/3217146429 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f180c7540 con 0x7f3f180c70d0 2023-06-23T00:40:26.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.385+0000 7f3f1d654700 1 -- 172.21.15.64:0/3217146429 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f1804c1b0 con 0x7f3f180c70d0 2023-06-23T00:40:26.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.385+0000 7f3efffff700 1 -- 172.21.15.64:0/3217146429 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f3f0c019030 con 0x7f3f180c70d0 2023-06-23T00:40:26.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.386+0000 7f3efffff700 1 --2- 172.21.15.64:0/3217146429 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3f04066e90 0x7f3f04069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.386+0000 7f3efffff700 1 -- 172.21.15.64:0/3217146429 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f3f0c085f70 con 0x7f3f180c70d0 2023-06-23T00:40:26.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.386+0000 7f3f17fff700 1 --2- 172.21.15.64:0/3217146429 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3f04066e90 0x7f3f04069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.386+0000 7f3f17fff700 1 --2- 172.21.15.64:0/3217146429 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7f3f04066e90 0x7f3f04069340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3f08000c00 tx=0x7f3f08009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:26.398 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.390+0000 7f3efffff700 1 -- 172.21.15.64:0/3217146429 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f3f0c0b7590 con 0x7f3f180c70d0 2023-06-23T00:40:26.633 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 1 closest matches: 2023-06-23T00:40:26.633 INFO:teuthology.orchestra.run.smithi064.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T00:40:26.633 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2023-06-23T00:40:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T00:40:26.637 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2023-06-23T00:40:26.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-23T00:40:26.760 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.756+0000 7faa7c496700 1 Processor -- start 2023-06-23T00:40:26.760 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.756+0000 7faa7c496700 1 -- start start 2023-06-23T00:40:26.761 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.757+0000 7faa7c496700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7faa740cd610 0x7faa74069e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.757+0000 7faa7c496700 1 -- --> v1:172.21.15.64:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740b1cd0 con 0x7faa740690f0 2023-06-23T00:40:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.757+0000 7faa7c496700 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740b1050 con 0x7faa740cd9e0 2023-06-23T00:40:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.757+0000 7faa7c496700 1 -- --> v2:172.21.15.64:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa740af880 con 0x7faa740cd610 2023-06-23T00:40:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.757+0000 7faa7a232700 1 --2- >> v2:172.21.15.64:3300/0 conn(0x7faa740cd610 0x7faa74069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.757+0000 7faa7aa33700 1 --1- >> v1:172.21.15.64:6789/0 conn(0x7faa740690f0 0x7faa740af170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.64:6789/0 says I am v1:172.21.15.64:43882/0 (socket says 172.21.15.64:43882) 2023-06-23T00:40:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.757+0000 7faa7aa33700 1 -- 172.21.15.64:0/1338541110 learned_addr learned my addr 172.21.15.64:0/1338541110 (peer_addr_for_me v1:172.21.15.64:0/0) 2023-06-23T00:40:26.762 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.758+0000 7faa7a232700 1 -- 172.21.15.64:0/1338541110 >> v1:172.21.15.86:6789/0 conn(0x7faa740cd9e0 legacy=0x7faa740aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T00:40:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.758+0000 7faa7a232700 1 -- 172.21.15.64:0/1338541110 >> v1:172.21.15.64:6789/0 conn(0x7faa740690f0 legacy=0x7faa740af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.758+0000 7faa79230700 1 -- 172.21.15.64:0/1338541110 <== mon.0 v1:172.21.15.64:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917272029 0 0) 0x7faa740b1cd0 con 0x7faa740690f0 2023-06-23T00:40:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.758+0000 7faa7a232700 1 -- 172.21.15.64:0/1338541110 --> v2:172.21.15.64:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa740ceb60 con 0x7faa740cd610 2023-06-23T00:40:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.758+0000 7faa7a232700 1 --2- 172.21.15.64:0/1338541110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa740cd610 0x7faa74069e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0098d0 tx=0x7faa6c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13040b169115158 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.758+0000 7faa79230700 1 -- 172.21.15.64:0/1338541110 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c009e70 con 0x7faa740cd610 2023-06-23T00:40:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.758+0000 7faa79230700 1 -- 172.21.15.64:0/1338541110 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6c005c10 con 0x7faa740cd610 2023-06-23T00:40:26.763 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.758+0000 7faa79230700 1 -- 172.21.15.64:0/1338541110 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c00b610 con 0x7faa740cd610 2023-06-23T00:40:26.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.758+0000 7faa7c496700 1 -- 172.21.15.64:0/1338541110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa740cd610 msgr2=0x7faa74069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.758+0000 7faa7c496700 1 --2- 172.21.15.64:0/1338541110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa740cd610 0x7faa74069e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0098d0 tx=0x7faa6c009120 comp rx=0 tx=0).stop 2023-06-23T00:40:26.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.759+0000 7faa7c496700 1 -- 172.21.15.64:0/1338541110 shutdown_connections 2023-06-23T00:40:26.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.759+0000 7faa7c496700 1 --2- 172.21.15.64:0/1338541110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa740cd610 0x7faa74069e30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.766 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.759+0000 7faa7c496700 1 -- 172.21.15.64:0/1338541110 >> 172.21.15.64:0/1338541110 conn(0x7faa74068cf0 msgr2=0x7faa740ce090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T00:40:26.767 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.759+0000 7faa7c496700 1 -- 172.21.15.64:0/1338541110 shutdown_connections 2023-06-23T00:40:26.767 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.759+0000 7faa7c496700 1 -- 172.21.15.64:0/1338541110 wait complete. 2023-06-23T00:40:26.767 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.760+0000 7faa7c496700 1 Processor -- start 2023-06-23T00:40:26.767 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.760+0000 7faa7c496700 1 -- start start 2023-06-23T00:40:26.768 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.760+0000 7faa7c496700 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa740690f0 0x7faa741537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.768 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa7c496700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa740cd610 0x7faa74154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.768 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa7c496700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa740cd9e0 0x7faa74155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.768 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa7c496700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7415afe0 con 0x7faa740cd610 2023-06-23T00:40:26.768 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa7c496700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7415a3d0 con 0x7faa740cd9e0 2023-06-23T00:40:26.768 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa7c496700 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7415a6b0 con 0x7faa740690f0 2023-06-23T00:40:26.768 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa79a31700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa740cd610 0x7faa74154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.768 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa79a31700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa740cd610 0x7faa74154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:41224/0 (socket says 172.21.15.64:41224) 2023-06-23T00:40:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa79a31700 1 -- 172.21.15.64:0/390061994 learned_addr learned my addr 172.21.15.64:0/390061994 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-06-23T00:40:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa7a232700 1 --2- 172.21.15.64:0/390061994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa740690f0 0x7faa741537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa7aa33700 1 --2- 172.21.15.64:0/390061994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa740cd9e0 0x7faa74155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa79a31700 1 -- 172.21.15.64:0/390061994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa740690f0 msgr2=0x7faa741537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa79a31700 1 --2- 172.21.15.64:0/390061994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa740690f0 0x7faa741537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa79a31700 1 -- 172.21.15.64:0/390061994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa740cd9e0 msgr2=0x7faa74155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T00:40:26.769 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa79a31700 1 --2- 172.21.15.64:0/390061994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa740cd9e0 0x7faa74155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T00:40:26.770 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa79a31700 1 -- 172.21.15.64:0/390061994 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6c0093c0 con 0x7faa740cd610 2023-06-23T00:40:26.770 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa7aa33700 1 --2- 172.21.15.64:0/390061994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa740cd9e0 0x7faa74155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T00:40:26.770 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa79a31700 1 --2- 172.21.15.64:0/390061994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa740cd610 0x7faa74154bf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faa6800b5e0 tx=0x7faa680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:26.770 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa727fc700 1 -- 172.21.15.64:0/390061994 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6800f510 con 0x7faa740cd610 2023-06-23T00:40:26.770 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa727fc700 1 -- 172.21.15.64:0/390061994 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6800f6b0 con 0x7faa740cd610 2023-06-23T00:40:26.770 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa7c496700 1 -- 172.21.15.64:0/390061994 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7415a990 con 0x7faa740cd610 2023-06-23T00:40:26.770 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa727fc700 1 -- 172.21.15.64:0/390061994 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6800fa70 con 0x7faa740cd610 2023-06-23T00:40:26.771 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.761+0000 7faa7c496700 1 -- 172.21.15.64:0/390061994 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa74159e30 con 0x7faa740cd610 2023-06-23T00:40:26.771 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.762+0000 7faa7c496700 1 -- 172.21.15.64:0/390061994 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7404c1b0 con 0x7faa740cd610 2023-06-23T00:40:26.771 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.763+0000 7faa727fc700 1 -- 172.21.15.64:0/390061994 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7faa6801a400 con 0x7faa740cd610 2023-06-23T00:40:26.772 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.763+0000 7faa727fc700 1 --2- 172.21.15.64:0/390061994 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7faa60066e90 0x7faa60069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T00:40:26.772 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.763+0000 7faa727fc700 1 -- 172.21.15.64:0/390061994 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7faa680843a0 con 0x7faa740cd610 2023-06-23T00:40:26.772 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.763+0000 7faa7a232700 1 --2- 172.21.15.64:0/390061994 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7faa60066e90 0x7faa60069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T00:40:26.772 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.764+0000 7faa7a232700 1 --2- 172.21.15.64:0/390061994 >> [v2:172.21.15.86:6832/117556,v1:172.21.15.86:6833/117556] conn(0x7faa60066e90 0x7faa60069340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009390 tx=0x7faa6c015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T00:40:26.772 INFO:teuthology.orchestra.run.smithi064.stderr:2023-06-23T00:40:26.767+0000 7faa727fc700 1 -- 172.21.15.64:0/390061994 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7faa68051300 con 0x7faa740cd610 2023-06-23T00:40:27.011 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 1 closest matches: 2023-06-23T00:40:27.011 INFO:teuthology.orchestra.run.smithi064.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T00:40:27.011 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2023-06-23T00:40:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T00:40:27.014 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T00:40:27.015 INFO:teuthology.misc:Shutting down mds daemons... 2023-06-23T00:40:27.016 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-23T00:40:27.017 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.025 INFO:tasks.ceph.mds.a:Stopped 2023-06-23T00:40:27.025 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-23T00:40:27.026 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.074 INFO:tasks.ceph.mds.c:Stopped 2023-06-23T00:40:27.075 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-23T00:40:27.076 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.086 INFO:tasks.ceph.mds.b:Stopped 2023-06-23T00:40:27.087 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-23T00:40:27.087 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.137 INFO:tasks.ceph.mds.d:Stopped 2023-06-23T00:40:27.138 INFO:teuthology.misc:Shutting down osd daemons... 2023-06-23T00:40:27.139 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-06-23T00:40:27.140 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.175 INFO:tasks.ceph.osd.0:Stopped 2023-06-23T00:40:27.175 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-06-23T00:40:27.176 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.240 INFO:tasks.ceph.osd.1:Stopped 2023-06-23T00:40:27.241 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-06-23T00:40:27.241 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.307 INFO:tasks.ceph.osd.2:Stopped 2023-06-23T00:40:27.308 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-06-23T00:40:27.309 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.373 INFO:tasks.ceph.osd.3:Stopped 2023-06-23T00:40:27.374 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-06-23T00:40:27.374 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.418 INFO:tasks.ceph.osd.4:Stopped 2023-06-23T00:40:27.419 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-06-23T00:40:27.419 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.462 INFO:tasks.ceph.osd.5:Stopped 2023-06-23T00:40:27.463 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-06-23T00:40:27.463 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.508 INFO:tasks.ceph.osd.6:Stopped 2023-06-23T00:40:27.509 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-06-23T00:40:27.510 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.546 INFO:tasks.ceph.osd.7:Stopped 2023-06-23T00:40:27.547 INFO:teuthology.misc:Shutting down mgr daemons... 2023-06-23T00:40:27.547 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-06-23T00:40:27.547 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.569 INFO:tasks.ceph.mgr.y:Stopped 2023-06-23T00:40:27.570 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-06-23T00:40:27.570 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.613 INFO:tasks.ceph.mgr.x:Stopped 2023-06-23T00:40:27.613 INFO:teuthology.misc:Shutting down mon daemons... 2023-06-23T00:40:27.614 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-06-23T00:40:27.614 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.624 INFO:tasks.ceph.mon.a:Stopped 2023-06-23T00:40:27.625 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-06-23T00:40:27.625 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.664 INFO:tasks.ceph.mon.b:Stopped 2023-06-23T00:40:27.664 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-06-23T00:40:27.665 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T00:40:27.715 INFO:tasks.ceph.mon.c:Stopped 2023-06-23T00:40:27.715 INFO:tasks.ceph:Checking cluster log for badness... 2023-06-23T00:40:27.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'bad backtrace' | egrep -v 'object missing on disk' | egrep -v 'error reading table object' | egrep -v 'error reading sessionmap' | egrep -v 'unmatched fragstat' | egrep -v 'unmatched rstat' | egrep -v 'was unreadable, recreating it now' | egrep -v 'Scrub error on inode' | egrep -v 'Metadata damage detected' | egrep -v MDS_FAILED | egrep -v MDS_DAMAGE | head -n 1 2023-06-23T00:40:27.749 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:40:27.749 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-06-23T00:40:28.910 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:40:28.911 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-06-23T00:40:29.054 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:40:29.055 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-06-23T00:40:29.242 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:40:29.242 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-06-23T00:40:29.346 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi086.front.sepia.ceph.com 2023-06-23T00:40:29.346 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-06-23T00:40:29.706 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi086.front.sepia.ceph.com 2023-06-23T00:40:29.706 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-06-23T00:40:29.816 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi086.front.sepia.ceph.com 2023-06-23T00:40:29.816 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-06-23T00:40:29.902 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi086.front.sepia.ceph.com 2023-06-23T00:40:29.902 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-06-23T00:40:30.021 INFO:tasks.ceph:Archiving mon data... 2023-06-23T00:40:30.023 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876/data/mon.a.tgz 2023-06-23T00:40:30.023 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2023-06-23T00:40:30.042 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.5OL1HlvolU 2023-06-23T00:40:30.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.5OL1HlvolU 2023-06-23T00:40:30.156 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.5OL1HlvolU 2023-06-23T00:40:30.242 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.5OL1HlvolU is 205KB 2023-06-23T00:40:30.306 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.5OL1HlvolU 2023-06-23T00:40:30.322 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876/data/mon.b.tgz 2023-06-23T00:40:30.322 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2023-06-23T00:40:30.338 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.U8e8Saug7S 2023-06-23T00:40:30.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.U8e8Saug7S 2023-06-23T00:40:30.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.U8e8Saug7S 2023-06-23T00:40:30.552 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.U8e8Saug7S is 287KB 2023-06-23T00:40:30.643 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.U8e8Saug7S 2023-06-23T00:40:30.658 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876/data/mon.c.tgz 2023-06-23T00:40:30.658 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2023-06-23T00:40:30.711 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.cELzp9ZqU6 2023-06-23T00:40:30.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.cELzp9ZqU6 2023-06-23T00:40:30.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.cELzp9ZqU6 2023-06-23T00:40:30.902 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.cELzp9ZqU6 is 287KB 2023-06-23T00:40:31.000 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.cELzp9ZqU6 2023-06-23T00:40:31.014 INFO:tasks.ceph:Cleaning ceph cluster... 2023-06-23T00:40:31.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-23T00:40:31.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-23T00:40:31.077 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-06-23T00:40:31.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-23T00:40:31.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-23T00:40:31.116 INFO:teuthology.orchestra.run.smithi064.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-23T00:40:31.143 INFO:teuthology.orchestra.run.smithi086.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-23T00:40:31.149 INFO:tasks.ceph:Archiving crash dumps... 2023-06-23T00:40:31.151 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876/remote/smithi064/crash 2023-06-23T00:40:31.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-23T00:40:31.192 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876/remote/smithi086/crash 2023-06-23T00:40:31.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-23T00:40:31.219 INFO:tasks.ceph:Shutting down logrotate 2023-06-23T00:40:31.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T00:40:31.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T00:40:31.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-23T00:40:31.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-23T00:40:31.369 INFO:tasks.ceph:Compressing logs... 2023-06-23T00:40:31.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T00:40:31.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T00:40:34.057 INFO:tasks.ceph:Archiving logs... 2023-06-23T00:40:34.058 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876/remote/smithi064/log 2023-06-23T00:40:34.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T00:40:34.315 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876/remote/smithi086/log 2023-06-23T00:40:34.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T00:40:34.571 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T00:40:34.658 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=66052aefc923485bb142771a599bba16 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T00:40:34.664 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-06-23T00:40:34.675 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-06-23T00:40:34.685 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-06-23T00:40:34.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T00:40:34.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T00:40:34.784 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-06-23T00:40:34.793 INFO:teuthology.task.clock:Checking final clock skew... 2023-06-23T00:40:34.794 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T00:40:34.796 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T00:40:34.808 INFO:teuthology.orchestra.run.smithi064.stderr:bash: ntpq: command not found 2023-06-23T00:40:34.810 INFO:teuthology.orchestra.run.smithi086.stderr:bash: ntpq: command not found 2023-06-23T00:40:34.811 INFO:teuthology.orchestra.run.smithi064.stdout:210 Number of sources = 4 2023-06-23T00:40:34.811 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T00:40:34.812 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2023-06-23T00:40:34.812 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +2473us[+2483us] +/- 118ms 2023-06-23T00:40:34.812 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 6 377 60 -1377us[-1366us] +/- 52ms 2023-06-23T00:40:34.812 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +465us[ +476us] +/- 73ms 2023-06-23T00:40:34.812 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:40:34.814 INFO:teuthology.orchestra.run.smithi086.stdout:210 Number of sources = 4 2023-06-23T00:40:34.814 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T00:40:34.814 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2023-06-23T00:40:34.814 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 60 +2456us[+2457us] +/- 118ms 2023-06-23T00:40:34.815 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -1397us[-1396us] +/- 52ms 2023-06-23T00:40:34.815 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +443us[ +443us] +/- 73ms 2023-06-23T00:40:34.815 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:40:34.815 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-06-23T00:40:34.824 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-06-23T00:40:34.825 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-06-23T00:40:34.836 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T00:40:34.945 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T00:40:35.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T00:40:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:40:35.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T00:40:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:40:35.094 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-06-23T00:40:35.104 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-06-23T00:40:35.112 INFO:teuthology.task.internal:Duration was 1203.186267 seconds 2023-06-23T00:40:35.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-06-23T00:40:35.122 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-06-23T00:40:35.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T00:40:35.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T00:40:35.414 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-06-23T00:40:35.415 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2023-06-23T00:40:35.415 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T00:40:35.435 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2023-06-23T00:40:35.436 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T00:40:35.454 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-06-23T00:40:35.455 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T00:40:35.478 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T00:40:35.532 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-06-23T00:40:35.533 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T00:40:35.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T00:40:35.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-06-23T00:40:35.657 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-06-23T00:40:35.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T00:40:35.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T00:40:35.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-06-23T00:40:35.724 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T00:40:35.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T00:40:35.749 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2023-06-23T00:40:35.772 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2023-06-23T00:40:35.786 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T00:40:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:40:35.818 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T00:40:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:40:35.840 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-06-23T00:40:35.850 INFO:teuthology.task.internal:Transferring archived files... 2023-06-23T00:40:35.851 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876/remote/smithi064 2023-06-23T00:40:35.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T00:40:35.893 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876/remote/smithi086 2023-06-23T00:40:35.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T00:40:35.926 INFO:teuthology.task.internal:Removing archive directory... 2023-06-23T00:40:35.926 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T00:40:35.929 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T00:40:35.975 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-06-23T00:40:35.987 INFO:teuthology.task.internal:Not uploading archives. 2023-06-23T00:40:35.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-06-23T00:40:35.999 INFO:teuthology.task.internal:Tidying up after the test... 2023-06-23T00:40:36.000 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T00:40:36.002 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T00:40:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 00:40 /home/ubuntu/cephtest 2023-06-23T00:40:36.029 INFO:teuthology.orchestra.run.smithi086.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 00:40 /home/ubuntu/cephtest 2023-06-23T00:40:36.031 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-06-23T00:40:36.044 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-06-23T00:40:36.070 INFO:teuthology.nuke:Checking targets against current locks 2023-06-23T00:40:36.099 DEBUG:teuthology.nuke:shortname: smithi064 2023-06-23T00:40:36.100 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T00:40:36.127 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:10:53.609891', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI35AffQKiJerBXQiexJsHNi/eRFM1jEN4uDgZt8gEo24uO9C7Qevh+1aUsEOcUiyFnWYA8rSQfZ+ukerdf7UXM='} 2023-06-23T00:40:36.130 DEBUG:teuthology.nuke:shortname: smithi086 2023-06-23T00:40:36.131 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T00:40:36.152 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311876', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:10:53.611194', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5jJYX/k8e04S5zh43QvZfHdNrNLdUA9OK/jgcrj2wjYuUsSN+K3Oo5ocRKvqb4CqxrGPGLjG2JmlD44PlF3+NxFjcFM/JdCe/A3Cj+/w2m09YrQUPEio0hSfLNCYGdAGmVdD/1U3urI8Z0eWooe3gQCj6E3+Qkh8/gVa/JQUYIrnsvI7OU/ZMSXrbhmBKy164mYKMTQBClxOV4Yb9kPf4CeNWK5SF5KrYu7n3nGO2CThcRomrpdBl/W8h0ZEbD3ALzmANcLgfJCZZuKdUM7xtCelXLEtvEqtinRTSelX6Xo7f859ZOUcIn3V1jQ6dpG6GxnF58IrXQI4llkJWfo00MvtlMZ7ZMgh3RRA97j6M+I8LrHayFNDE2Rk6eCOgfnQ5BLvu93X7VVmSBmUBWCwEpHOaMSkPnH+lTZkvTdjnyn8capN1C8Rpnr6H/27RTccw20khijHhhdvrUh2oisPBw4vGLD1WdfW2bpTIQXQggaV3S/Cvy+rVKBPzp2ZIkTE='} 2023-06-23T00:40:36.188 INFO:teuthology.orchestra.console:Power off smithi064 2023-06-23T00:40:36.190 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T00:40:36.216 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T00:40:36.243 INFO:teuthology.orchestra.console:Power off smithi086 2023-06-23T00:40:36.244 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T00:40:36.274 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T00:40:40.262 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T00:40:40.302 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T00:40:44.380 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T00:40:44.414 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T00:40:44.592 INFO:teuthology.orchestra.console:Power off for smithi064 completed 2023-06-23T00:40:44.627 INFO:teuthology.orchestra.console:Power off for smithi086 completed 2023-06-23T00:40:44.732 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} duration: 1203.186266899109 failure_reason: 'Command failed on smithi064 with status 22: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=66052aefc923485bb142771a599bba16 status: fail success: false 2023-06-23T00:40:44.733 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T00:40:44.826 INFO:teuthology.run:FAIL