2023-06-23T00:42:22.991 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-23T00:42:22.992 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi --owner scheduled_yuriw@teuthology --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} --archive /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908 --verbose -- /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908/orig.config.yaml 2023-06-23T00:42:23.039 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T00:42:23.128 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908 branch: wip-yuri4-testing-2023-06-21-0846-quincy description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} email: yweinste@redhat.com first_in_suite: false job_id: '7311908' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-06-21-0846-quincy ceph: cephfs: session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - SLOW_OPS - slow request sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 check-counter: counters: mds: - mds.dir_split install: ceph: flavor: default sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c kclient: mntopts: - ms_mode=legacy - wsync selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-06-21-0846-quincy sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 9037 sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2023-06-21-0846-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c targets: smithi046.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN1AP+f9En/qWCTR517bW3K2TEIx74WvyU0+UStEBwjA4HOA+Dk6quiGXlPiyJiQLebEZpT7q6txD3U7TWt2yw7R0ASa4rIaQQ5eD17stmLN8AbYAqo3HnMFe+/arNslnB2u3mO3A4Cv5GvnQDxSnRCyJPgprPV8Ohsle9rfQcnF2iFigaAJluuHxMhEgfq4dSQ+cWZO0y/2AF3wfchXJXnPZf6ePR+mNyqYkrBnVoZy3pXlDhI5mRQN0WxcEFWcekaYi53sf6v1RDamz50QMKoOqdWD955PL8h66luT2Gm7nM/qOqtvHCC8tbFxlqsIpQgJ/39LjZNQXplFpms1AaJqRMk7gUAY9RSG9JdkW7oqglr0N7GzemCKoNuOGYptFRWsUkM5T+mqujhRPIwgZGFAoreeRHmf8vre8oYz/gy+qsYe2hk5eHuIIB0j/xdYu5ayfkOCqXFa6pyejJNI/SE7kpRDexHtONnhmnDK9p6PRD+rEEAnB8vJ4e5QlfoEk= smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJrLrBzXWVmQ3L1uZgHMn/snozkRm3Cwl6ZcJjgQX3uwpqIHAHSgbISAqsC2Hq8cVRoNT/YeDmx0bg9WR8bTItw= smithi131.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnDHs2ZAbaoFqu4S0M5u5Efu8EYNCAAOCkrnxRvUtNDTXZ9v7+gc4/D2UifAgtA2MWSXlTfVfWLbQ8giOkgulaOgsGSQ1PJUul1xefDvNiGsbr0gajnQ/SMB68P4R31eukuYf6BiMIZs6onNLqJzotTtMDV5hhSs6LakIbxzUMeyiW95TImavGgv2hF6esNLyefFLi3TQTl8gUma6LBeaOzTu8VMevCnrzyFd7XcTvd42vlY77zzPtTZbCDPocI59botf9ajpTe8Mkbwn+JHxCkGAiZWuzxM0ZNZ0D2OxR7XYbqHYOQxQLYlxMXIr13y4X087xo9rGbzPGXAXtUVfwZnQd3KqYqu5VgIcvqX6nlGvHGvtVwhs4ATbtzV9VHubyiJC6u0ZjYy7amMLIrLNFYIgDzHXRkGBm0xu9ftVvrJfJtkz/VQiEOEujTZBuCL5/6SqRApvNI+9K/W3iA5tzC/y1plO2MEZzmpW3b3B2mDVaLmbfCpbObPltoDhljSc= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - check-counter: {} - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 076bbebc42a14f7d568aaa78eabb0038327bcb23 timestamp: 2023-06-22_15:25:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-06-23T00:42:23.129 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa; will attempt to use it 2023-06-23T00:42:23.130 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks 2023-06-23T00:42:23.130 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-06-23T00:42:23.132 INFO:teuthology.task.internal:Checking packages... 2023-06-23T00:42:23.172 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '42ddc46e6e6582d8e29a5a982a81886fe028be2c' 2023-06-23T00:42:23.173 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-06-23T00:42:23.173 INFO:teuthology.packaging:ref: None 2023-06-23T00:42:23.173 INFO:teuthology.packaging:tag: None 2023-06-23T00:42:23.173 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T00:42:23.174 INFO:teuthology.packaging:sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:42:23.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T00:42:23.398 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-719.g42ddc46e 2023-06-23T00:42:23.399 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-06-23T00:42:23.405 INFO:teuthology.task.internal:no buildpackages task found 2023-06-23T00:42:23.405 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-06-23T00:42:23.420 INFO:teuthology.task.internal:Saving configuration 2023-06-23T00:42:23.437 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-06-23T00:42:23.444 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T00:42:23.473 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:37:21.158050', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOuSVlyVDKFO9HbMlHfxO47oFnDppy2OvqahmFQzTYyh63j7juqhGanXE4QqfTr4IVax3MGKol04J9u4KenePHc='} 2023-06-23T00:42:23.501 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:37:21.156577', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJrLrBzXWVmQ3L1uZgHMn/snozkRm3Cwl6ZcJjgQX3uwpqIHAHSgbISAqsC2Hq8cVRoNT/YeDmx0bg9WR8bTItw='} 2023-06-23T00:42:23.525 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:37:21.159316', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqsabipcdc9+TwUDH5nA5oDedVwo8S77YKze1oWp3+2W3BJ1BEGtr2kKJctiqXnfYfRlQqm2HzQLvMPvdCCkh0='} 2023-06-23T00:42:23.525 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-06-23T00:42:23.532 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-06-23T00:42:23.533 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-06-23T00:42:23.533 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-06-23T00:42:23.533 INFO:teuthology.run_tasks:Running task console_log... 2023-06-23T00:42:23.645 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6b66dd00d0>, signals=[15]) 2023-06-23T00:42:23.646 INFO:teuthology.run_tasks:Running task internal.connect... 2023-06-23T00:42:23.653 INFO:teuthology.task.internal:Opening connections... 2023-06-23T00:42:23.653 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2023-06-23T00:42:23.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:42:23.729 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T00:42:23.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:42:23.796 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2023-06-23T00:42:23.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:42:23.867 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-06-23T00:42:23.873 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2023-06-23T00:42:23.892 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2023-06-23T00:42:23.892 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2023-06-23T00:42:23.949 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="Red Hat Enterprise Linux" 2023-06-23T00:42:23.949 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="8.4 (Ootpa)" 2023-06-23T00:42:23.949 INFO:teuthology.orchestra.run.smithi046.stdout:ID="rhel" 2023-06-23T00:42:23.949 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="fedora" 2023-06-23T00:42:23.950 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="8.4" 2023-06-23T00:42:23.950 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el8" 2023-06-23T00:42:23.950 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-23T00:42:23.950 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2023-06-23T00:42:23.950 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-23T00:42:23.950 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://www.redhat.com/" 2023-06-23T00:42:23.950 INFO:teuthology.orchestra.run.smithi046.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-23T00:42:23.951 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T00:42:23.951 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:42:23.951 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T00:42:23.951 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-23T00:42:23.951 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-23T00:42:23.951 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-23T00:42:23.952 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2023-06-23T00:42:23.980 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2023-06-23T00:42:23.997 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2023-06-23T00:42:23.998 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2023-06-23T00:42:24.053 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="Red Hat Enterprise Linux" 2023-06-23T00:42:24.053 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="8.4 (Ootpa)" 2023-06-23T00:42:24.053 INFO:teuthology.orchestra.run.smithi092.stdout:ID="rhel" 2023-06-23T00:42:24.053 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="fedora" 2023-06-23T00:42:24.053 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="8.4" 2023-06-23T00:42:24.053 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el8" 2023-06-23T00:42:24.054 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-23T00:42:24.054 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2023-06-23T00:42:24.054 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-23T00:42:24.054 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://www.redhat.com/" 2023-06-23T00:42:24.054 INFO:teuthology.orchestra.run.smithi092.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-23T00:42:24.054 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T00:42:24.054 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:42:24.055 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T00:42:24.055 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-23T00:42:24.055 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-23T00:42:24.055 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-23T00:42:24.055 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2023-06-23T00:42:24.088 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2023-06-23T00:42:24.107 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2023-06-23T00:42:24.107 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2023-06-23T00:42:24.163 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="Red Hat Enterprise Linux" 2023-06-23T00:42:24.163 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="8.4 (Ootpa)" 2023-06-23T00:42:24.164 INFO:teuthology.orchestra.run.smithi131.stdout:ID="rhel" 2023-06-23T00:42:24.164 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="fedora" 2023-06-23T00:42:24.164 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="8.4" 2023-06-23T00:42:24.164 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el8" 2023-06-23T00:42:24.164 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-23T00:42:24.164 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2023-06-23T00:42:24.164 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-23T00:42:24.165 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://www.redhat.com/" 2023-06-23T00:42:24.165 INFO:teuthology.orchestra.run.smithi131.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-23T00:42:24.165 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T00:42:24.165 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:42:24.165 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T00:42:24.165 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-23T00:42:24.166 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-23T00:42:24.166 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-23T00:42:24.166 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2023-06-23T00:42:24.197 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-06-23T00:42:24.210 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-06-23T00:42:24.217 INFO:teuthology.task.internal:Checking for old test directory... 2023-06-23T00:42:24.218 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2023-06-23T00:42:24.221 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2023-06-23T00:42:24.224 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2023-06-23T00:42:24.240 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-06-23T00:42:24.246 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-06-23T00:42:24.246 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2023-06-23T00:42:24.278 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2023-06-23T00:42:24.281 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2023-06-23T00:42:24.292 INFO:teuthology.orchestra.run.smithi046.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-23T00:42:24.294 INFO:teuthology.orchestra.run.smithi092.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-23T00:42:24.298 INFO:teuthology.orchestra.run.smithi131.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-23T00:42:24.299 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-06-23T00:42:24.384 INFO:teuthology.run_tasks:Running task kernel... 2023-06-23T00:42:24.406 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-06-23T00:42:24.407 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-06-23T00:42:24.407 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T00:42:24.408 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T00:42:24.409 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T00:42:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:42:24.425 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2023-06-23T00:42:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:42:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:42:24.427 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2023-06-23T00:42:24.428 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2023-06-23T00:42:24.481 INFO:teuthology.orchestra.run.smithi092.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:42:24.481 INFO:teuthology.task.kernel:Running kernel on smithi092: 4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:42:24.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum install -y kernel 2023-06-23T00:42:24.483 INFO:teuthology.orchestra.run.smithi131.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:42:24.484 INFO:teuthology.task.kernel:Running kernel on smithi131: 4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:42:24.484 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum install -y kernel 2023-06-23T00:42:24.485 INFO:teuthology.orchestra.run.smithi046.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:42:24.486 INFO:teuthology.task.kernel:Running kernel on smithi046: 4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:42:24.486 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2023-06-23T00:42:25.899 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T00:42:25.900 INFO:teuthology.orchestra.run.smithi092.stdout:Unable to read consumer identity 2023-06-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2023-06-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi046.stdout:Unable to read consumer identity 2023-06-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2023-06-23T00:42:26.032 INFO:teuthology.orchestra.run.smithi131.stdout:Unable to read consumer identity 2023-06-23T00:42:29.596 INFO:teuthology.orchestra.run.smithi092.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-06-23T00:42:29.628 INFO:teuthology.orchestra.run.smithi131.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-06-23T00:42:29.638 INFO:teuthology.orchestra.run.smithi046.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T00:42:32.520 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 16 MB 00:02 2023-06-23T00:42:32.528 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 5.5 MB/s | 16 MB 00:02 2023-06-23T00:42:32.551 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 5.4 MB/s | 16 MB 00:02 2023-06-23T00:42:35.732 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2023-06-23T00:42:35.745 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2023-06-23T00:42:36.026 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2023-06-23T00:42:36.436 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-23T00:42:36.436 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-23T00:42:36.437 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-23T00:42:36.465 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-23T00:42:36.466 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-23T00:42:36.466 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-23T00:42:36.595 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-06-23T00:42:36.596 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-06-23T00:42:36.596 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-06-23T00:42:36.612 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-06-23T00:42:36.613 INFO:teuthology.orchestra.run.smithi131.stdout:Nothing to do. 2023-06-23T00:42:36.613 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-06-23T00:42:36.699 DEBUG:teuthology.orchestra.run.smithi092:> echo no | sudo yum reinstall kernel || true 2023-06-23T00:42:36.721 DEBUG:teuthology.orchestra.run.smithi131:> echo no | sudo yum reinstall kernel || true 2023-06-23T00:42:36.803 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-23T00:42:36.803 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-23T00:42:36.804 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-23T00:42:36.952 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2023-06-23T00:42:36.953 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2023-06-23T00:42:36.953 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2023-06-23T00:42:37.044 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T00:42:37.045 INFO:teuthology.orchestra.run.smithi092.stdout:Unable to read consumer identity 2023-06-23T00:42:37.060 DEBUG:teuthology.orchestra.run.smithi046:> echo no | sudo yum reinstall kernel || true 2023-06-23T00:42:37.066 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2023-06-23T00:42:37.067 INFO:teuthology.orchestra.run.smithi131.stdout:Unable to read consumer identity 2023-06-23T00:42:37.333 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 12:42:35 AM UTC. 2023-06-23T00:42:37.352 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 12:42:35 AM UTC. 2023-06-23T00:42:37.426 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2023-06-23T00:42:37.426 INFO:teuthology.orchestra.run.smithi046.stdout:Unable to read consumer identity 2023-06-23T00:42:37.463 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:42:37.464 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:37.464 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:37.492 INFO:teuthology.orchestra.run.smithi131.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:42:37.492 INFO:teuthology.orchestra.run.smithi131.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:37.493 INFO:teuthology.orchestra.run.smithi131.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:37.493 INFO:teuthology.orchestra.run.smithi092.stderr:Error: No packages marked for reinstall. 2023-06-23T00:42:37.542 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum reinstall -y kernel || true 2023-06-23T00:42:37.551 INFO:teuthology.orchestra.run.smithi131.stderr:Error: No packages marked for reinstall. 2023-06-23T00:42:37.602 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum reinstall -y kernel || true 2023-06-23T00:42:37.736 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:01 ago on Fri 23 Jun 2023 12:42:36 AM UTC. 2023-06-23T00:42:37.884 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:42:37.884 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:37.884 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:37.885 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T00:42:37.885 INFO:teuthology.orchestra.run.smithi092.stdout:Unable to read consumer identity 2023-06-23T00:42:37.911 INFO:teuthology.orchestra.run.smithi046.stderr:Error: No packages marked for reinstall. 2023-06-23T00:42:37.944 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2023-06-23T00:42:37.945 INFO:teuthology.orchestra.run.smithi131.stdout:Unable to read consumer identity 2023-06-23T00:42:37.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum reinstall -y kernel || true 2023-06-23T00:42:38.166 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:03 ago on Fri 23 Jun 2023 12:42:35 AM UTC. 2023-06-23T00:42:38.227 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:00:03 ago on Fri 23 Jun 2023 12:42:35 AM UTC. 2023-06-23T00:42:38.288 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:42:38.288 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:38.289 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:38.323 INFO:teuthology.orchestra.run.smithi092.stderr:Error: No packages marked for reinstall. 2023-06-23T00:42:38.327 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2023-06-23T00:42:38.327 INFO:teuthology.orchestra.run.smithi046.stdout:Unable to read consumer identity 2023-06-23T00:42:38.363 INFO:teuthology.orchestra.run.smithi131.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:42:38.363 INFO:teuthology.orchestra.run.smithi131.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:38.363 INFO:teuthology.orchestra.run.smithi131.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:38.373 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:42:38.379 INFO:teuthology.orchestra.run.smithi131.stderr:Error: No packages marked for reinstall. 2023-06-23T00:42:38.432 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:42:38.498 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.498 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.499 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.499 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T00:42:38.499 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T00:42:38.499 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-06-23T00:42:38.500 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T00:42:38.500 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:42:38.557 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.557 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.557 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.557 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T00:42:38.558 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T00:42:38.558 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2023-06-23T00:42:38.558 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi131.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T00:42:38.558 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:42:38.623 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 12:42:36 AM UTC. 2023-06-23T00:42:38.625 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.625 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -qi grub2-tools 2023-06-23T00:42:38.653 INFO:teuthology.orchestra.run.smithi092.stdout:Name : grub2-tools 2023-06-23T00:42:38.653 INFO:teuthology.orchestra.run.smithi092.stdout:Epoch : 1 2023-06-23T00:42:38.653 INFO:teuthology.orchestra.run.smithi092.stdout:Version : 2.02 2023-06-23T00:42:38.654 INFO:teuthology.orchestra.run.smithi092.stdout:Release : 99.el8_4.1 2023-06-23T00:42:38.654 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: x86_64 2023-06-23T00:42:38.654 INFO:teuthology.orchestra.run.smithi092.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-23T00:42:38.654 INFO:teuthology.orchestra.run.smithi092.stdout:Group : System Environment/Base 2023-06-23T00:42:38.655 INFO:teuthology.orchestra.run.smithi092.stdout:Size : 9264631 2023-06-23T00:42:38.655 INFO:teuthology.orchestra.run.smithi092.stdout:License : GPLv3+ 2023-06-23T00:42:38.655 INFO:teuthology.orchestra.run.smithi092.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-23T00:42:38.655 INFO:teuthology.orchestra.run.smithi092.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-23T00:42:38.655 INFO:teuthology.orchestra.run.smithi092.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-23T00:42:38.655 INFO:teuthology.orchestra.run.smithi092.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-23T00:42:38.655 INFO:teuthology.orchestra.run.smithi092.stdout:Relocations : (not relocatable) 2023-06-23T00:42:38.655 INFO:teuthology.orchestra.run.smithi092.stdout:Packager : Red Hat, Inc. 2023-06-23T00:42:38.656 INFO:teuthology.orchestra.run.smithi092.stdout:Vendor : Red Hat, Inc. 2023-06-23T00:42:38.656 INFO:teuthology.orchestra.run.smithi092.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T00:42:38.656 INFO:teuthology.orchestra.run.smithi092.stdout:Summary : Support tools for GRUB. 2023-06-23T00:42:38.656 INFO:teuthology.orchestra.run.smithi092.stdout:Description : 2023-06-23T00:42:38.656 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:42:38.656 INFO:teuthology.orchestra.run.smithi092.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T00:42:38.656 INFO:teuthology.orchestra.run.smithi092.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T00:42:38.657 INFO:teuthology.orchestra.run.smithi092.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T00:42:38.657 INFO:teuthology.orchestra.run.smithi092.stdout:hardware devices. 2023-06-23T00:42:38.657 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:42:38.658 INFO:teuthology.orchestra.run.smithi092.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T00:42:38.659 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T00:42:38.659 INFO:teuthology.task.kernel:Updating grub on smithi092 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T00:42:38.683 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.683 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.684 DEBUG:teuthology.orchestra.run.smithi131:> sudo rpm -qi grub2-tools 2023-06-23T00:42:38.710 INFO:teuthology.orchestra.run.smithi131.stdout:Name : grub2-tools 2023-06-23T00:42:38.710 INFO:teuthology.orchestra.run.smithi131.stdout:Epoch : 1 2023-06-23T00:42:38.710 INFO:teuthology.orchestra.run.smithi131.stdout:Version : 2.02 2023-06-23T00:42:38.711 INFO:teuthology.orchestra.run.smithi131.stdout:Release : 99.el8_4.1 2023-06-23T00:42:38.711 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: x86_64 2023-06-23T00:42:38.711 INFO:teuthology.orchestra.run.smithi131.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-23T00:42:38.711 INFO:teuthology.orchestra.run.smithi131.stdout:Group : System Environment/Base 2023-06-23T00:42:38.711 INFO:teuthology.orchestra.run.smithi131.stdout:Size : 9264631 2023-06-23T00:42:38.711 INFO:teuthology.orchestra.run.smithi131.stdout:License : GPLv3+ 2023-06-23T00:42:38.711 INFO:teuthology.orchestra.run.smithi131.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-23T00:42:38.712 INFO:teuthology.orchestra.run.smithi131.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-23T00:42:38.712 INFO:teuthology.orchestra.run.smithi131.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-23T00:42:38.712 INFO:teuthology.orchestra.run.smithi131.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-23T00:42:38.712 INFO:teuthology.orchestra.run.smithi131.stdout:Relocations : (not relocatable) 2023-06-23T00:42:38.712 INFO:teuthology.orchestra.run.smithi131.stdout:Packager : Red Hat, Inc. 2023-06-23T00:42:38.712 INFO:teuthology.orchestra.run.smithi131.stdout:Vendor : Red Hat, Inc. 2023-06-23T00:42:38.713 INFO:teuthology.orchestra.run.smithi131.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T00:42:38.713 INFO:teuthology.orchestra.run.smithi131.stdout:Summary : Support tools for GRUB. 2023-06-23T00:42:38.713 INFO:teuthology.orchestra.run.smithi131.stdout:Description : 2023-06-23T00:42:38.713 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:42:38.713 INFO:teuthology.orchestra.run.smithi131.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T00:42:38.713 INFO:teuthology.orchestra.run.smithi131.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T00:42:38.713 INFO:teuthology.orchestra.run.smithi131.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T00:42:38.714 INFO:teuthology.orchestra.run.smithi131.stdout:hardware devices. 2023-06-23T00:42:38.714 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:42:38.714 INFO:teuthology.orchestra.run.smithi131.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T00:42:38.715 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T00:42:38.715 INFO:teuthology.task.kernel:Updating grub on smithi131 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.715 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T00:42:38.766 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:42:38.767 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:38.767 INFO:teuthology.orchestra.run.smithi046.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:42:38.779 INFO:teuthology.orchestra.run.smithi046.stderr:Error: No packages marked for reinstall. 2023-06-23T00:42:38.830 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:42:38.963 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.963 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.964 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:38.964 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T00:42:38.964 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T00:42:38.964 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-06-23T00:42:38.964 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T00:42:38.965 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:42:39.094 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:39.094 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:39.095 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -qi grub2-tools 2023-06-23T00:42:39.122 INFO:teuthology.orchestra.run.smithi046.stdout:Name : grub2-tools 2023-06-23T00:42:39.123 INFO:teuthology.orchestra.run.smithi046.stdout:Epoch : 1 2023-06-23T00:42:39.123 INFO:teuthology.orchestra.run.smithi046.stdout:Version : 2.02 2023-06-23T00:42:39.123 INFO:teuthology.orchestra.run.smithi046.stdout:Release : 99.el8_4.1 2023-06-23T00:42:39.123 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: x86_64 2023-06-23T00:42:39.123 INFO:teuthology.orchestra.run.smithi046.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-23T00:42:39.123 INFO:teuthology.orchestra.run.smithi046.stdout:Group : System Environment/Base 2023-06-23T00:42:39.124 INFO:teuthology.orchestra.run.smithi046.stdout:Size : 9264631 2023-06-23T00:42:39.124 INFO:teuthology.orchestra.run.smithi046.stdout:License : GPLv3+ 2023-06-23T00:42:39.124 INFO:teuthology.orchestra.run.smithi046.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-23T00:42:39.124 INFO:teuthology.orchestra.run.smithi046.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-23T00:42:39.124 INFO:teuthology.orchestra.run.smithi046.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-23T00:42:39.124 INFO:teuthology.orchestra.run.smithi046.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-23T00:42:39.124 INFO:teuthology.orchestra.run.smithi046.stdout:Relocations : (not relocatable) 2023-06-23T00:42:39.125 INFO:teuthology.orchestra.run.smithi046.stdout:Packager : Red Hat, Inc. 2023-06-23T00:42:39.125 INFO:teuthology.orchestra.run.smithi046.stdout:Vendor : Red Hat, Inc. 2023-06-23T00:42:39.125 INFO:teuthology.orchestra.run.smithi046.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T00:42:39.125 INFO:teuthology.orchestra.run.smithi046.stdout:Summary : Support tools for GRUB. 2023-06-23T00:42:39.125 INFO:teuthology.orchestra.run.smithi046.stdout:Description : 2023-06-23T00:42:39.125 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:42:39.125 INFO:teuthology.orchestra.run.smithi046.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T00:42:39.125 INFO:teuthology.orchestra.run.smithi046.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T00:42:39.126 INFO:teuthology.orchestra.run.smithi046.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T00:42:39.126 INFO:teuthology.orchestra.run.smithi046.stdout:hardware devices. 2023-06-23T00:42:39.126 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:42:39.126 INFO:teuthology.orchestra.run.smithi046.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T00:42:39.127 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T00:42:39.127 INFO:teuthology.task.kernel:Updating grub on smithi046 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:39.127 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T00:42:39.888 INFO:teuthology.orchestra.run.smithi131.stderr:Generating grub configuration file ... 2023-06-23T00:42:39.916 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2023-06-23T00:42:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2023-06-23T00:42:41.803 INFO:teuthology.orchestra.run.smithi131.stderr:done 2023-06-23T00:42:41.804 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2023-06-23T00:42:41.819 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.rXiqTR3G7U 2023-06-23T00:42:41.820 DEBUG:teuthology.orchestra.run.smithi131:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rXiqTR3G7U 2023-06-23T00:42:41.890 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.rXiqTR3G7U 2023-06-23T00:42:42.049 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.rXiqTR3G7U is 6KB 2023-06-23T00:42:42.096 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.rXiqTR3G7U 2023-06-23T00:42:42.109 DEBUG:teuthology.orchestra.run.smithi131:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T00:42:42.181 INFO:teuthology.orchestra.run.smithi131.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-23T00:42:42.181 INFO:teuthology.orchestra.run.smithi131.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-23T00:42:42.181 INFO:teuthology.orchestra.run.smithi131.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-23T00:42:42.181 INFO:teuthology.orchestra.run.smithi131.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-23T00:42:42.183 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:42.319 INFO:teuthology.orchestra.run.smithi092.stderr:done 2023-06-23T00:42:42.321 DEBUG:teuthology.orchestra.run.smithi131:> sudo shutdown -r now 2023-06-23T00:42:42.328 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2023-06-23T00:42:42.343 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.v04WKgUW63 2023-06-23T00:42:42.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.v04WKgUW63 2023-06-23T00:42:42.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.v04WKgUW63 2023-06-23T00:42:42.570 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.v04WKgUW63 is 6KB 2023-06-23T00:42:42.588 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.v04WKgUW63 2023-06-23T00:42:42.602 DEBUG:teuthology.orchestra.run.smithi092:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T00:42:42.677 INFO:teuthology.orchestra.run.smithi092.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-23T00:42:42.678 INFO:teuthology.orchestra.run.smithi092.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-23T00:42:42.678 INFO:teuthology.orchestra.run.smithi092.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-23T00:42:42.678 INFO:teuthology.orchestra.run.smithi092.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-23T00:42:42.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:42.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2023-06-23T00:42:43.141 INFO:teuthology.orchestra.run.smithi046.stderr:done 2023-06-23T00:42:43.142 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2023-06-23T00:42:43.156 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.MWus2g8xsc 2023-06-23T00:42:43.157 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MWus2g8xsc 2023-06-23T00:42:43.240 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.MWus2g8xsc 2023-06-23T00:42:43.399 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.MWus2g8xsc is 6KB 2023-06-23T00:42:43.417 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.MWus2g8xsc 2023-06-23T00:42:43.431 DEBUG:teuthology.orchestra.run.smithi046:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T00:42:43.501 INFO:teuthology.orchestra.run.smithi046.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-23T00:42:43.501 INFO:teuthology.orchestra.run.smithi046.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-23T00:42:43.502 INFO:teuthology.orchestra.run.smithi046.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-23T00:42:43.502 INFO:teuthology.orchestra.run.smithi046.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-23T00:42:43.503 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:42:43.655 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2023-06-23T00:43:12.352 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2023-06-23T00:43:12.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-06-23T00:43:12.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:43:12.835 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-06-23T00:43:12.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-06-23T00:43:12.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:43:13.658 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-06-23T00:43:13.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-06-23T00:43:13.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:43:31.312 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-06-23T00:43:48.208 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2023-06-23T00:44:01.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-06-23T00:44:01.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:44:12.365 DEBUG:teuthology.orchestra.remote:timed out 2023-06-23T00:44:18.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-06-23T00:44:18.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:44:19.700 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-06-23T00:44:34.272 DEBUG:teuthology.orchestra.run.smithi046:> true 2023-06-23T00:44:34.594 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-06-23T00:44:34.595 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-23T00:44:34.595 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2023-06-23T00:44:34.611 INFO:teuthology.orchestra.run.smithi046.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-23T00:44:34.612 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:44:34.613 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T00:44:34.613 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-23T00:44:35.614 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-06-23T00:44:35.615 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-23T00:44:35.838 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2023-06-23T00:44:35.863 DEBUG:teuthology.parallel:result is None 2023-06-23T00:44:42.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-06-23T00:44:42.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:44:42.623 DEBUG:teuthology.orchestra.run.smithi131:> true 2023-06-23T00:44:42.877 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-06-23T00:44:42.878 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-23T00:44:42.878 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2023-06-23T00:44:42.893 INFO:teuthology.orchestra.run.smithi131.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-23T00:44:42.894 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:44:42.894 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T00:44:42.895 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-23T00:44:43.896 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-06-23T00:44:43.896 DEBUG:teuthology.orchestra.run.smithi131:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-23T00:44:43.979 INFO:teuthology.orchestra.run.smithi131.stdout:ttyS1 2023-06-23T00:44:43.994 DEBUG:teuthology.parallel:result is None 2023-06-23T00:44:49.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-06-23T00:44:49.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:44:50.041 DEBUG:teuthology.orchestra.run.smithi092:> true 2023-06-23T00:44:50.326 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-06-23T00:44:50.327 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-23T00:44:50.327 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2023-06-23T00:44:50.343 INFO:teuthology.orchestra.run.smithi092.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-23T00:44:50.343 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:44:50.344 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T00:44:50.344 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-23T00:44:51.345 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-06-23T00:44:51.345 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-23T00:44:51.437 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2023-06-23T00:44:51.448 DEBUG:teuthology.parallel:result is None 2023-06-23T00:44:51.449 INFO:teuthology.run_tasks:Running task internal.base... 2023-06-23T00:44:51.456 INFO:teuthology.task.internal:Creating test directory... 2023-06-23T00:44:51.456 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T00:44:51.460 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T00:44:51.462 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T00:44:51.479 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-06-23T00:44:51.486 INFO:teuthology.run_tasks:Running task internal.archive... 2023-06-23T00:44:51.498 INFO:teuthology.task.internal:Creating archive directory... 2023-06-23T00:44:51.499 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T00:44:51.519 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T00:44:51.521 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T00:44:51.782 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-06-23T00:44:51.790 INFO:teuthology.task.internal:Enabling coredump saving... 2023-06-23T00:44:51.791 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T00:44:51.793 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T00:44:51.795 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T00:44:51.825 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:44:51.833 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:44:51.836 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:44:51.843 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:44:51.847 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:44:51.855 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:44:51.856 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-06-23T00:44:51.863 INFO:teuthology.task.internal:Configuring sudo... 2023-06-23T00:44:51.864 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T00:44:51.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T00:44:51.893 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T00:44:51.920 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-06-23T00:44:51.929 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-06-23T00:44:51.930 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T00:44:51.960 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T00:44:51.962 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T00:44:51.977 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:44:52.060 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:44:52.100 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:44:52.186 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:44:52.224 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T00:44:52.224 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T00:44:52.289 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:44:52.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:44:52.374 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:44:52.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:44:52.496 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T00:44:52.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T00:44:52.561 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:44:52.609 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:44:52.648 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:44:52.734 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:44:52.773 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-06-23T00:44:52.773 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T00:44:52.838 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2023-06-23T00:44:52.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2023-06-23T00:44:52.842 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2023-06-23T00:44:52.895 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T00:44:52.907 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T00:44:52.932 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T00:44:53.329 INFO:teuthology.run_tasks:Running task internal.timer... 2023-06-23T00:44:53.337 INFO:teuthology.task.internal:Starting timer... 2023-06-23T00:44:53.337 INFO:teuthology.run_tasks:Running task pcp... 2023-06-23T00:44:53.351 INFO:teuthology.run_tasks:Running task selinux... 2023-06-23T00:44:53.360 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-06-23T00:44:53.361 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2023-06-23T00:44:53.440 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: [ OK ] 2023-06-23T00:44:53.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2023-06-23T00:44:53.509 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: [ OK ] 2023-06-23T00:44:53.511 DEBUG:teuthology.orchestra.run.smithi131:> sudo service auditd rotate 2023-06-23T00:44:53.565 INFO:teuthology.orchestra.run.smithi131.stdout:Rotating logs: [ OK ] 2023-06-23T00:44:53.567 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-06-23T00:44:53.567 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2023-06-23T00:44:53.590 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2023-06-23T00:44:53.591 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2023-06-23T00:44:53.616 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2023-06-23T00:44:53.616 DEBUG:teuthology.orchestra.run.smithi131:> /usr/sbin/getenforce 2023-06-23T00:44:53.641 INFO:teuthology.orchestra.run.smithi131.stdout:Permissive 2023-06-23T00:44:53.642 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi092.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi131.front.sepia.ceph.com': 'permissive'} 2023-06-23T00:44:53.642 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T00:44:53.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:44:53.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T00:44:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:44:53.697 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T00:44:53.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:44:53.724 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-06-23T00:44:53.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2023-06-23T00:44:53.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2023-06-23T00:44:53.776 DEBUG:teuthology.orchestra.run.smithi131:> sudo /usr/sbin/setenforce permissive 2023-06-23T00:44:53.802 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-06-23T00:44:53.813 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-06-23T00:44:53.813 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-06-23T00:44:53.834 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-06-23T00:44:53.836 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi092.front.sepia.ceph.com,smithi131.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-06-23T00:53:04.708 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi131.front.sepia.ceph.com')] 2023-06-23T00:53:04.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-06-23T00:53:04.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:53:04.781 DEBUG:teuthology.orchestra.run.smithi046:> true 2023-06-23T00:53:04.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-06-23T00:53:04.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:53:04.958 DEBUG:teuthology.orchestra.run.smithi092:> true 2023-06-23T00:53:04.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-06-23T00:53:04.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:53:05.053 DEBUG:teuthology.orchestra.run.smithi131:> true 2023-06-23T00:53:05.121 INFO:teuthology.run_tasks:Running task clock... 2023-06-23T00:53:05.140 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-06-23T00:53:05.141 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T00:53:05.142 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T00:53:05.145 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T00:53:05.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T00:53:05.150 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T00:53:05.151 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T00:53:05.176 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T00:53:05.181 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T00:53:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T00:53:05.192 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T00:53:05.196 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T00:53:05.207 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T00:53:05.227 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2023-06-23T00:53:05.235 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2023-06-23T00:53:05.240 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2023-06-23T00:53:05.242 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: ntpd: command not found 2023-06-23T00:53:05.246 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2023-06-23T00:53:05.256 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T00:53:05.261 INFO:teuthology.orchestra.run.smithi131.stdout:506 Cannot talk to daemon 2023-06-23T00:53:05.262 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T00:53:05.267 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T00:53:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T00:53:05.274 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T00:53:05.280 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T00:53:05.340 INFO:teuthology.orchestra.run.smithi092.stderr:bash: ntpq: command not found 2023-06-23T00:53:05.342 INFO:teuthology.orchestra.run.smithi092.stdout:210 Number of sources = 4 2023-06-23T00:53:05.343 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T00:53:05.343 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2023-06-23T00:53:05.343 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.343 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.343 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.343 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.344 INFO:teuthology.orchestra.run.smithi131.stderr:bash: ntpq: command not found 2023-06-23T00:53:05.346 INFO:teuthology.orchestra.run.smithi131.stdout:210 Number of sources = 4 2023-06-23T00:53:05.346 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T00:53:05.347 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2023-06-23T00:53:05.347 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.347 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.347 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.347 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.348 INFO:teuthology.orchestra.run.smithi046.stderr:bash: ntpq: command not found 2023-06-23T00:53:05.351 INFO:teuthology.orchestra.run.smithi046.stdout:210 Number of sources = 4 2023-06-23T00:53:05.351 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T00:53:05.351 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2023-06-23T00:53:05.351 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.352 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.352 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.352 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T00:53:05.352 INFO:teuthology.run_tasks:Running task pexec... 2023-06-23T00:53:05.361 INFO:teuthology.task.pexec:Executing custom commands... 2023-06-23T00:53:05.362 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi046.front.sepia.ceph.com 2023-06-23T00:53:05.362 DEBUG:teuthology.orchestra.run.smithi046:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-06-23T00:53:05.363 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T00:53:05.363 DEBUG:teuthology.orchestra.run.smithi092:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-06-23T00:53:05.363 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi131.front.sepia.ceph.com 2023-06-23T00:53:05.364 DEBUG:teuthology.orchestra.run.smithi131:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-06-23T00:53:05.775 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:05.785 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:05.802 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:06.658 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:49 ago on Fri 23 Jun 2023 12:51:17 AM UTC. 2023-06-23T00:53:06.680 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:45 ago on Fri 23 Jun 2023 12:51:21 AM UTC. 2023-06-23T00:53:06.721 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:34 ago on Fri 23 Jun 2023 12:51:32 AM UTC. 2023-06-23T00:53:08.786 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-06-23T00:53:08.787 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-06-23T00:53:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2023-06-23T00:53:08.787 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-06-23T00:53:08.787 INFO:teuthology.orchestra.run.smithi092.stdout:Resetting modules: 2023-06-23T00:53:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: container-tools 2023-06-23T00:53:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:53:08.788 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-06-23T00:53:08.788 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-06-23T00:53:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:53:08.809 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-06-23T00:53:08.810 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-06-23T00:53:08.810 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2023-06-23T00:53:08.810 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-06-23T00:53:08.810 INFO:teuthology.orchestra.run.smithi131.stdout:Resetting modules: 2023-06-23T00:53:08.810 INFO:teuthology.orchestra.run.smithi131.stdout: container-tools 2023-06-23T00:53:08.810 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:53:08.811 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-06-23T00:53:08.811 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2023-06-23T00:53:08.811 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:53:08.836 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2023-06-23T00:53:08.837 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2023-06-23T00:53:08.838 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2023-06-23T00:53:08.838 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2023-06-23T00:53:08.838 INFO:teuthology.orchestra.run.smithi046.stdout:Resetting modules: 2023-06-23T00:53:08.838 INFO:teuthology.orchestra.run.smithi046.stdout: container-tools 2023-06-23T00:53:08.838 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:53:08.838 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2023-06-23T00:53:08.839 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2023-06-23T00:53:08.839 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:53:09.726 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:52 ago on Fri 23 Jun 2023 12:51:17 AM UTC. 2023-06-23T00:53:09.747 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:48 ago on Fri 23 Jun 2023 12:51:21 AM UTC. 2023-06-23T00:53:09.808 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:37 ago on Fri 23 Jun 2023 12:51:32 AM UTC. 2023-06-23T00:53:11.659 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-06-23T00:53:11.960 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-06-23T00:53:11.971 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2023-06-23T00:53:12.104 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:12.409 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:12.441 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:12.934 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:51 ago on Fri 23 Jun 2023 12:51:21 AM UTC. 2023-06-23T00:53:13.239 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:55 ago on Fri 23 Jun 2023 12:51:17 AM UTC. 2023-06-23T00:53:13.314 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:40 ago on Fri 23 Jun 2023 12:51:32 AM UTC. 2023-06-23T00:53:14.908 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-06-23T00:53:14.957 INFO:teuthology.orchestra.run.smithi131.stdout:========================================================================================================= 2023-06-23T00:53:14.957 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2023-06-23T00:53:14.957 INFO:teuthology.orchestra.run.smithi131.stdout:========================================================================================================= 2023-06-23T00:53:14.957 INFO:teuthology.orchestra.run.smithi131.stdout:Installing group/module packages: 2023-06-23T00:53:14.958 INFO:teuthology.orchestra.run.smithi131.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2023-06-23T00:53:14.958 INFO:teuthology.orchestra.run.smithi131.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2023-06-23T00:53:14.958 INFO:teuthology.orchestra.run.smithi131.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2023-06-23T00:53:14.958 INFO:teuthology.orchestra.run.smithi131.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-06-23T00:53:14.958 INFO:teuthology.orchestra.run.smithi131.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2023-06-23T00:53:14.958 INFO:teuthology.orchestra.run.smithi131.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2023-06-23T00:53:14.958 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-06-23T00:53:14.958 INFO:teuthology.orchestra.run.smithi131.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-06-23T00:53:14.959 INFO:teuthology.orchestra.run.smithi131.stdout:Installing module profiles: 2023-06-23T00:53:14.959 INFO:teuthology.orchestra.run.smithi131.stdout: container-tools/common 2023-06-23T00:53:14.959 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:53:14.959 INFO:teuthology.orchestra.run.smithi131.stdout:Enabling module streams: 2023-06-23T00:53:14.959 INFO:teuthology.orchestra.run.smithi131.stdout: container-tools rhel8 2023-06-23T00:53:14.959 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:53:14.959 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-06-23T00:53:14.960 INFO:teuthology.orchestra.run.smithi131.stdout:========================================================================================================= 2023-06-23T00:53:14.960 INFO:teuthology.orchestra.run.smithi131.stdout:Install 7 Packages 2023-06-23T00:53:14.960 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:53:14.960 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 16 M 2023-06-23T00:53:14.961 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 57 M 2023-06-23T00:53:14.961 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-06-23T00:53:15.006 INFO:teuthology.orchestra.run.smithi131.stdout:(1/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 4.0 MB/s | 192 kB 00:00 2023-06-23T00:53:15.010 INFO:teuthology.orchestra.run.smithi131.stdout:(2/7): yajl-2.1.0-10.el8.x86_64.rpm 800 kB/s | 41 kB 00:00 2023-06-23T00:53:15.020 INFO:teuthology.orchestra.run.smithi131.stdout:(3/7): cockpit-podman-32-2.module+el8.4.0+11990 6.6 MB/s | 410 kB 00:00 2023-06-23T00:53:15.036 INFO:teuthology.orchestra.run.smithi131.stdout:(4/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 517 kB/s | 16 kB 00:00 2023-06-23T00:53:15.068 INFO:teuthology.orchestra.run.smithi131.stdout:(5/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 1.6 MB/s | 51 kB 00:00 2023-06-23T00:53:15.126 INFO:teuthology.orchestra.run.smithi131.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 61 MB/s | 7.0 MB 00:00 2023-06-23T00:53:15.164 INFO:teuthology.orchestra.run.smithi131.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 58 MB/s | 8.2 MB 00:00 2023-06-23T00:53:15.164 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-06-23T00:53:15.164 INFO:teuthology.orchestra.run.smithi131.stdout:Total 78 MB/s | 16 MB 00:00 2023-06-23T00:53:15.168 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-06-23T00:53:15.189 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-06-23T00:53:15.190 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-06-23T00:53:15.262 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-06-23T00:53:15.309 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================================= 2023-06-23T00:53:15.310 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-06-23T00:53:15.310 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================================= 2023-06-23T00:53:15.310 INFO:teuthology.orchestra.run.smithi092.stdout:Installing group/module packages: 2023-06-23T00:53:15.310 INFO:teuthology.orchestra.run.smithi092.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2023-06-23T00:53:15.310 INFO:teuthology.orchestra.run.smithi092.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2023-06-23T00:53:15.310 INFO:teuthology.orchestra.run.smithi092.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2023-06-23T00:53:15.310 INFO:teuthology.orchestra.run.smithi092.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-06-23T00:53:15.311 INFO:teuthology.orchestra.run.smithi092.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2023-06-23T00:53:15.311 INFO:teuthology.orchestra.run.smithi092.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2023-06-23T00:53:15.311 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-06-23T00:53:15.311 INFO:teuthology.orchestra.run.smithi092.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-06-23T00:53:15.311 INFO:teuthology.orchestra.run.smithi092.stdout:Installing module profiles: 2023-06-23T00:53:15.311 INFO:teuthology.orchestra.run.smithi092.stdout: container-tools/common 2023-06-23T00:53:15.311 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:53:15.312 INFO:teuthology.orchestra.run.smithi092.stdout:Enabling module streams: 2023-06-23T00:53:15.312 INFO:teuthology.orchestra.run.smithi092.stdout: container-tools rhel8 2023-06-23T00:53:15.312 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:53:15.313 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-06-23T00:53:15.313 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================================= 2023-06-23T00:53:15.313 INFO:teuthology.orchestra.run.smithi092.stdout:Install 7 Packages 2023-06-23T00:53:15.313 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:53:15.314 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 16 M 2023-06-23T00:53:15.314 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 57 M 2023-06-23T00:53:15.314 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-06-23T00:53:15.343 INFO:teuthology.orchestra.run.smithi092.stdout:(1/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 6.0 MB/s | 192 kB 00:00 2023-06-23T00:53:15.349 INFO:teuthology.orchestra.run.smithi092.stdout:(2/7): cockpit-podman-32-2.module+el8.4.0+11990 11 MB/s | 410 kB 00:00 2023-06-23T00:53:15.351 INFO:teuthology.orchestra.run.smithi092.stdout:(3/7): yajl-2.1.0-10.el8.x86_64.rpm 1.0 MB/s | 41 kB 00:00 2023-06-23T00:53:15.359 INFO:teuthology.orchestra.run.smithi092.stdout:(4/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 979 kB/s | 16 kB 00:00 2023-06-23T00:53:15.375 INFO:teuthology.orchestra.run.smithi092.stdout:(5/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 3.2 MB/s | 51 kB 00:00 2023-06-23T00:53:15.421 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2023-06-23T00:53:15.448 INFO:teuthology.orchestra.run.smithi092.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 71 MB/s | 7.0 MB 00:00 2023-06-23T00:53:15.475 INFO:teuthology.orchestra.run.smithi046.stdout:========================================================================================================= 2023-06-23T00:53:15.475 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2023-06-23T00:53:15.475 INFO:teuthology.orchestra.run.smithi046.stdout:========================================================================================================= 2023-06-23T00:53:15.475 INFO:teuthology.orchestra.run.smithi046.stdout:Installing group/module packages: 2023-06-23T00:53:15.476 INFO:teuthology.orchestra.run.smithi046.stdout: buildah x86_64 1.21.4-2.module+el8.4.0+12050+ef972f71 rhel-8-for-x86_64-appstream-rpms 8.2 M 2023-06-23T00:53:15.476 INFO:teuthology.orchestra.run.smithi046.stdout: cockpit-podman noarch 32-2.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 410 k 2023-06-23T00:53:15.476 INFO:teuthology.orchestra.run.smithi046.stdout: crun x86_64 0.20.1-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 192 k 2023-06-23T00:53:15.476 INFO:teuthology.orchestra.run.smithi046.stdout: skopeo x86_64 1:1.3.1-5.module+el8.4.0+11990+22932769 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-06-23T00:53:15.476 INFO:teuthology.orchestra.run.smithi046.stdout: toolbox noarch 0.0.8-1.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 16 k 2023-06-23T00:53:15.476 INFO:teuthology.orchestra.run.smithi046.stdout: udica noarch 0.2.4-2.module+el8.4.0+11822+6cc1e7d7 rhel-8-for-x86_64-appstream-rpms 51 k 2023-06-23T00:53:15.476 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2023-06-23T00:53:15.477 INFO:teuthology.orchestra.run.smithi046.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-06-23T00:53:15.477 INFO:teuthology.orchestra.run.smithi046.stdout:Installing module profiles: 2023-06-23T00:53:15.477 INFO:teuthology.orchestra.run.smithi046.stdout: container-tools/common 2023-06-23T00:53:15.477 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:53:15.477 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling module streams: 2023-06-23T00:53:15.477 INFO:teuthology.orchestra.run.smithi046.stdout: container-tools rhel8 2023-06-23T00:53:15.477 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:53:15.477 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2023-06-23T00:53:15.478 INFO:teuthology.orchestra.run.smithi046.stdout:========================================================================================================= 2023-06-23T00:53:15.478 INFO:teuthology.orchestra.run.smithi046.stdout:Install 7 Packages 2023-06-23T00:53:15.478 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:53:15.479 INFO:teuthology.orchestra.run.smithi092.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 66 MB/s | 8.2 MB 00:00 2023-06-23T00:53:15.479 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-06-23T00:53:15.479 INFO:teuthology.orchestra.run.smithi092.stdout:Total 96 MB/s | 16 MB 00:00 2023-06-23T00:53:15.479 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 16 M 2023-06-23T00:53:15.479 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 57 M 2023-06-23T00:53:15.479 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2023-06-23T00:53:15.482 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-06-23T00:53:15.499 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-06-23T00:53:15.499 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-06-23T00:53:15.517 INFO:teuthology.orchestra.run.smithi046.stdout:(1/7): cockpit-podman-32-2.module+el8.4.0+11990 10 MB/s | 410 kB 00:00 2023-06-23T00:53:15.518 INFO:teuthology.orchestra.run.smithi046.stdout:(2/7): yajl-2.1.0-10.el8.x86_64.rpm 1.0 MB/s | 41 kB 00:00 2023-06-23T00:53:15.532 INFO:teuthology.orchestra.run.smithi046.stdout:(3/7): crun-0.20.1-1.module+el8.4.0+11822+6cc1e 3.5 MB/s | 192 kB 00:00 2023-06-23T00:53:15.533 INFO:teuthology.orchestra.run.smithi046.stdout:(4/7): toolbox-0.0.8-1.module+el8.4.0+11822+6cc 1.1 MB/s | 16 kB 00:00 2023-06-23T00:53:15.546 INFO:teuthology.orchestra.run.smithi046.stdout:(5/7): udica-0.2.4-2.module+el8.4.0+11822+6cc1e 3.4 MB/s | 51 kB 00:00 2023-06-23T00:53:15.642 INFO:teuthology.orchestra.run.smithi046.stdout:(6/7): skopeo-1.3.1-5.module+el8.4.0+11990+2293 57 MB/s | 7.0 MB 00:00 2023-06-23T00:53:15.677 INFO:teuthology.orchestra.run.smithi046.stdout:(7/7): buildah-1.21.4-2.module+el8.4.0+12050+ef 56 MB/s | 8.2 MB 00:00 2023-06-23T00:53:15.677 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2023-06-23T00:53:15.677 INFO:teuthology.orchestra.run.smithi046.stdout:Total 79 MB/s | 16 MB 00:00 2023-06-23T00:53:15.684 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2023-06-23T00:53:15.702 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2023-06-23T00:53:15.703 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2023-06-23T00:53:15.705 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-06-23T00:53:15.709 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-06-23T00:53:16.059 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-06-23T00:53:16.064 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-06-23T00:53:16.099 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-06-23T00:53:16.284 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2023-06-23T00:53:16.289 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2023-06-23T00:53:16.297 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-23T00:53:16.477 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-06-23T00:53:16.505 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-23T00:53:16.665 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-23T00:53:16.750 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2023-06-23T00:53:16.840 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-23T00:53:16.964 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-23T00:53:17.166 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-23T00:53:17.395 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 3/7 2023-06-23T00:53:17.761 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 3/7 2023-06-23T00:53:18.086 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 3/7 2023-06-23T00:53:18.178 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 4/7 2023-06-23T00:53:18.356 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-23T00:53:18.508 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 6/7 2023-06-23T00:53:18.587 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 4/7 2023-06-23T00:53:18.661 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-23T00:53:18.774 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-23T00:53:18.927 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-23T00:53:18.927 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-23T00:53:18.928 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-23T00:53:18.928 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 3/7 2023-06-23T00:53:18.928 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 4/7 2023-06-23T00:53:18.928 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-23T00:53:18.928 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 6/7 2023-06-23T00:53:18.930 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 4/7 2023-06-23T00:53:18.961 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 6/7 2023-06-23T00:53:19.082 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-23T00:53:19.122 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-23T00:53:19.259 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 6/7 2023-06-23T00:53:19.380 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-23T00:53:19.383 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-23T00:53:19.383 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-23T00:53:19.384 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-23T00:53:19.384 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 3/7 2023-06-23T00:53:19.384 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 4/7 2023-06-23T00:53:19.384 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-23T00:53:19.384 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 6/7 2023-06-23T00:53:19.665 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 7/7 2023-06-23T00:53:19.665 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 1/7 2023-06-23T00:53:19.666 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2/7 2023-06-23T00:53:19.666 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cockpit-podman-32-2.module+el8.4.0+11990+22932769.no 3/7 2023-06-23T00:53:19.666 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 4/7 2023-06-23T00:53:19.666 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_6 5/7 2023-06-23T00:53:19.666 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_6 6/7 2023-06-23T00:53:20.007 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2023-06-23T00:53:20.007 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:58 ago on Fri 23 Jun 2023 12:51:21 AM UTC. 2023-06-23T00:53:20.551 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2023-06-23T00:53:20.551 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:02:03 ago on Fri 23 Jun 2023 12:51:17 AM UTC. 2023-06-23T00:53:20.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 7/7 2023-06-23T00:53:20.826 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:48 ago on Fri 23 Jun 2023 12:51:32 AM UTC. 2023-06-23T00:53:22.016 INFO:teuthology.orchestra.run.smithi131.stdout:Installed products updated. 2023-06-23T00:53:22.117 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:53:22.117 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-06-23T00:53:22.117 INFO:teuthology.orchestra.run.smithi131.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2023-06-23T00:53:22.117 INFO:teuthology.orchestra.run.smithi131.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2023-06-23T00:53:22.117 INFO:teuthology.orchestra.run.smithi131.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2023-06-23T00:53:22.118 INFO:teuthology.orchestra.run.smithi131.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2023-06-23T00:53:22.118 INFO:teuthology.orchestra.run.smithi131.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-23T00:53:22.118 INFO:teuthology.orchestra.run.smithi131.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-23T00:53:22.118 INFO:teuthology.orchestra.run.smithi131.stdout: yajl-2.1.0-10.el8.x86_64 2023-06-23T00:53:22.118 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:53:22.118 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-06-23T00:53:22.241 DEBUG:teuthology.parallel:result is None 2023-06-23T00:53:22.609 INFO:teuthology.orchestra.run.smithi092.stdout:Installed products updated. 2023-06-23T00:53:22.718 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:53:22.718 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-06-23T00:53:22.719 INFO:teuthology.orchestra.run.smithi092.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2023-06-23T00:53:22.719 INFO:teuthology.orchestra.run.smithi092.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2023-06-23T00:53:22.719 INFO:teuthology.orchestra.run.smithi092.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2023-06-23T00:53:22.719 INFO:teuthology.orchestra.run.smithi092.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2023-06-23T00:53:22.720 INFO:teuthology.orchestra.run.smithi092.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-23T00:53:22.720 INFO:teuthology.orchestra.run.smithi092.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-23T00:53:22.721 INFO:teuthology.orchestra.run.smithi092.stdout: yajl-2.1.0-10.el8.x86_64 2023-06-23T00:53:22.721 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:53:22.721 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-06-23T00:53:22.860 DEBUG:teuthology.parallel:result is None 2023-06-23T00:53:22.916 INFO:teuthology.orchestra.run.smithi046.stdout:Installed products updated. 2023-06-23T00:53:23.026 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:53:23.026 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2023-06-23T00:53:23.026 INFO:teuthology.orchestra.run.smithi046.stdout: buildah-1.21.4-2.module+el8.4.0+12050+ef972f71.x86_64 2023-06-23T00:53:23.026 INFO:teuthology.orchestra.run.smithi046.stdout: cockpit-podman-32-2.module+el8.4.0+11990+22932769.noarch 2023-06-23T00:53:23.027 INFO:teuthology.orchestra.run.smithi046.stdout: crun-0.20.1-1.module+el8.4.0+11822+6cc1e7d7.x86_64 2023-06-23T00:53:23.027 INFO:teuthology.orchestra.run.smithi046.stdout: skopeo-1:1.3.1-5.module+el8.4.0+11990+22932769.x86_64 2023-06-23T00:53:23.027 INFO:teuthology.orchestra.run.smithi046.stdout: toolbox-0.0.8-1.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-23T00:53:23.027 INFO:teuthology.orchestra.run.smithi046.stdout: udica-0.2.4-2.module+el8.4.0+11822+6cc1e7d7.noarch 2023-06-23T00:53:23.027 INFO:teuthology.orchestra.run.smithi046.stdout: yajl-2.1.0-10.el8.x86_64 2023-06-23T00:53:23.027 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:53:23.028 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2023-06-23T00:53:23.154 DEBUG:teuthology.parallel:result is None 2023-06-23T00:53:23.154 INFO:teuthology.run_tasks:Running task install... 2023-06-23T00:53:23.166 DEBUG:teuthology.task.install:project ceph 2023-06-23T00:53:23.167 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}} 2023-06-23T00:53:23.168 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'} 2023-06-23T00:53:23.168 INFO:teuthology.task.install:Using flavor: default 2023-06-23T00:53:23.177 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-06-23T00:53:23.178 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-06-23T00:53:23.178 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T00:53:23.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:53:23.181 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T00:53:23.185 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:53:23.188 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T00:53:23.189 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:53:23.350 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T00:53:23.350 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T00:53:23.413 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T00:53:23.414 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T00:53:23.438 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T00:53:23.438 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T00:53:23.463 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T00:53:23.463 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T00:53:23.464 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T00:53:23.495 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T00:53:23.496 DEBUG:teuthology.orchestra.run.smithi092:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T00:53:23.561 DEBUG:teuthology.orchestra.run.smithi092:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T00:53:23.579 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T00:53:23.579 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T00:53:23.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T00:53:23.602 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T00:53:23.603 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-06-23T00:53:23.603 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T00:53:23.607 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T00:53:23.607 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T00:53:23.629 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T00:53:23.629 DEBUG:teuthology.orchestra.run.smithi131:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T00:53:23.638 DEBUG:teuthology.orchestra.run.smithi092:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T00:53:23.673 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T00:53:23.695 DEBUG:teuthology.orchestra.run.smithi131:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T00:53:23.705 INFO:teuthology.orchestra.run.smithi092.stdout:check_obsoletes = 1 2023-06-23T00:53:23.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum clean all 2023-06-23T00:53:23.751 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T00:53:23.773 DEBUG:teuthology.orchestra.run.smithi131:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T00:53:23.823 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2023-06-23T00:53:23.823 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2023-06-23T00:53:23.836 INFO:teuthology.orchestra.run.smithi131.stdout:check_obsoletes = 1 2023-06-23T00:53:23.839 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum clean all 2023-06-23T00:53:24.095 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:24.229 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:24.236 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:24.409 INFO:teuthology.orchestra.run.smithi092.stdout:44 files removed 2023-06-23T00:53:24.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T00:53:24.543 INFO:teuthology.orchestra.run.smithi131.stdout:44 files removed 2023-06-23T00:53:24.553 INFO:teuthology.orchestra.run.smithi046.stdout:44 files removed 2023-06-23T00:53:24.594 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T00:53:24.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T00:53:24.809 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:24.944 INFO:teuthology.orchestra.run.smithi131.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:24.974 INFO:teuthology.orchestra.run.smithi046.stdout:Updating Subscription Management repositories. 2023-06-23T00:53:25.602 INFO:teuthology.orchestra.run.smithi092.stdout:ceph packages for x86_64 233 kB/s | 78 kB 00:00 2023-06-23T00:53:25.718 INFO:teuthology.orchestra.run.smithi131.stdout:ceph packages for x86_64 253 kB/s | 78 kB 00:00 2023-06-23T00:53:25.729 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 284 kB/s | 78 kB 00:00 2023-06-23T00:53:25.944 INFO:teuthology.orchestra.run.smithi092.stdout:ceph noarch packages 30 kB/s | 9.2 kB 00:00 2023-06-23T00:53:26.058 INFO:teuthology.orchestra.run.smithi131.stdout:ceph noarch packages 31 kB/s | 9.2 kB 00:00 2023-06-23T00:53:26.081 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 30 kB/s | 9.2 kB 00:00 2023-06-23T00:53:26.275 INFO:teuthology.orchestra.run.smithi092.stdout:ceph source packages 4.8 kB/s | 1.4 kB 00:00 2023-06-23T00:53:26.356 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 6.0 kB/s | 1.4 kB 00:00 2023-06-23T00:53:26.391 INFO:teuthology.orchestra.run.smithi131.stdout:ceph source packages 4.8 kB/s | 1.4 kB 00:00 2023-06-23T00:53:28.526 INFO:teuthology.orchestra.run.smithi092.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2023-06-23T00:53:28.814 INFO:teuthology.orchestra.run.smithi046.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T00:53:28.867 INFO:teuthology.orchestra.run.smithi131.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T00:53:30.497 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 16 MB 00:01 2023-06-23T00:53:31.057 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 16 MB 00:02 2023-06-23T00:53:31.163 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 6.9 MB/s | 16 MB 00:02 2023-06-23T00:53:33.731 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2023-06-23T00:53:34.086 INFO:teuthology.orchestra.run.smithi092.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 120 MB/s | 33 MB 00:00 2023-06-23T00:53:34.380 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2023-06-23T00:53:34.446 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 5.1 MB/s | 24 kB 00:00 2023-06-23T00:53:34.732 INFO:teuthology.orchestra.run.smithi131.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 121 MB/s | 33 MB 00:00 2023-06-23T00:53:34.850 INFO:teuthology.orchestra.run.smithi046.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 103 MB/s | 33 MB 00:00 2023-06-23T00:53:39.209 INFO:teuthology.orchestra.run.smithi092.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 126 MB/s | 38 MB 00:00 2023-06-23T00:53:39.869 INFO:teuthology.orchestra.run.smithi131.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 122 MB/s | 38 MB 00:00 2023-06-23T00:53:40.320 INFO:teuthology.orchestra.run.smithi046.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 110 MB/s | 38 MB 00:00 2023-06-23T00:53:47.555 INFO:teuthology.orchestra.run.smithi092.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T00:53:47.556 INFO:teuthology.orchestra.run.smithi092.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T00:53:47.566 INFO:teuthology.orchestra.run.smithi092.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-23T00:53:47.567 INFO:teuthology.orchestra.run.smithi092.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-23T00:53:47.568 INFO:teuthology.orchestra.run.smithi092.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-23T00:53:47.569 INFO:teuthology.orchestra.run.smithi092.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T00:53:47.571 INFO:teuthology.orchestra.run.smithi092.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T00:53:47.573 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-23T00:53:47.575 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-23T00:53:48.070 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-06-23T00:53:48.081 INFO:teuthology.orchestra.run.smithi092.stdout:====================================================================================================================== 2023-06-23T00:53:48.082 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-06-23T00:53:48.082 INFO:teuthology.orchestra.run.smithi092.stdout:====================================================================================================================== 2023-06-23T00:53:48.082 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-06-23T00:53:48.082 INFO:teuthology.orchestra.run.smithi092.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-23T00:53:48.082 INFO:teuthology.orchestra.run.smithi092.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T00:53:48.083 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T00:53:48.083 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T00:53:48.083 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T00:53:48.083 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T00:53:48.083 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T00:53:48.083 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T00:53:48.083 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T00:53:48.084 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T00:53:48.084 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T00:53:48.084 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T00:53:48.084 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T00:53:48.084 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T00:53:48.084 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T00:53:48.084 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T00:53:48.085 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-23T00:53:48.085 INFO:teuthology.orchestra.run.smithi092.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-23T00:53:48.085 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-23T00:53:48.085 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T00:53:48.085 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T00:53:48.085 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T00:53:48.085 INFO:teuthology.orchestra.run.smithi092.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-23T00:53:48.086 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-23T00:53:48.086 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-23T00:53:48.086 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T00:53:48.086 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T00:53:48.086 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T00:53:48.086 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T00:53:48.086 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T00:53:48.087 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T00:53:48.087 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T00:53:48.087 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-23T00:53:48.087 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2023-06-23T00:53:48.087 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T00:53:48.088 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T00:53:48.088 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-06-23T00:53:48.088 INFO:teuthology.orchestra.run.smithi092.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-23T00:53:48.088 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T00:53:48.088 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T00:53:48.088 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T00:53:48.089 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T00:53:48.089 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T00:53:48.089 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T00:53:48.089 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T00:53:48.089 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T00:53:48.089 INFO:teuthology.orchestra.run.smithi092.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-23T00:53:48.089 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T00:53:48.090 INFO:teuthology.orchestra.run.smithi092.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-23T00:53:48.090 INFO:teuthology.orchestra.run.smithi092.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-23T00:53:48.090 INFO:teuthology.orchestra.run.smithi092.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-23T00:53:48.090 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T00:53:48.090 INFO:teuthology.orchestra.run.smithi092.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-23T00:53:48.090 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-23T00:53:48.090 INFO:teuthology.orchestra.run.smithi092.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T00:53:48.090 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-23T00:53:48.091 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-23T00:53:48.091 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T00:53:48.091 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-23T00:53:48.091 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T00:53:48.091 INFO:teuthology.orchestra.run.smithi092.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-23T00:53:48.091 INFO:teuthology.orchestra.run.smithi092.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-23T00:53:48.092 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T00:53:48.092 INFO:teuthology.orchestra.run.smithi092.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-23T00:53:48.092 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-23T00:53:48.092 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-23T00:53:48.092 INFO:teuthology.orchestra.run.smithi092.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-23T00:53:48.092 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-23T00:53:48.093 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-23T00:53:48.093 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-23T00:53:48.093 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-23T00:53:48.093 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T00:53:48.093 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-23T00:53:48.093 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T00:53:48.093 INFO:teuthology.orchestra.run.smithi092.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T00:53:48.093 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T00:53:48.094 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T00:53:48.094 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T00:53:48.094 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T00:53:48.094 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T00:53:48.094 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T00:53:48.094 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T00:53:48.094 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T00:53:48.095 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T00:53:48.095 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-23T00:53:48.095 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-23T00:53:48.095 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-23T00:53:48.095 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T00:53:48.095 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T00:53:48.095 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T00:53:48.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-23T00:53:48.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-23T00:53:48.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T00:53:48.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T00:53:48.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-23T00:53:48.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-23T00:53:48.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-23T00:53:48.096 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T00:53:48.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T00:53:48.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-23T00:53:48.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-23T00:53:48.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-23T00:53:48.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-23T00:53:48.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T00:53:48.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T00:53:48.097 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T00:53:48.098 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T00:53:48.098 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-23T00:53:48.098 INFO:teuthology.orchestra.run.smithi092.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T00:53:48.098 INFO:teuthology.orchestra.run.smithi092.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T00:53:48.098 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T00:53:48.098 INFO:teuthology.orchestra.run.smithi092.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T00:53:48.098 INFO:teuthology.orchestra.run.smithi092.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T00:53:48.099 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T00:53:48.099 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T00:53:48.099 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T00:53:48.099 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-23T00:53:48.099 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T00:53:48.099 INFO:teuthology.orchestra.run.smithi092.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-23T00:53:48.099 INFO:teuthology.orchestra.run.smithi092.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T00:53:48.099 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-23T00:53:48.100 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-23T00:53:48.100 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:53:48.100 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-06-23T00:53:48.100 INFO:teuthology.orchestra.run.smithi092.stdout:====================================================================================================================== 2023-06-23T00:53:48.100 INFO:teuthology.orchestra.run.smithi092.stdout:Install 118 Packages 2023-06-23T00:53:48.100 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 2 Packages 2023-06-23T00:53:48.100 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:53:48.101 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 201 M 2023-06-23T00:53:48.101 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-06-23T00:53:48.233 INFO:teuthology.orchestra.run.smithi131.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T00:53:48.236 INFO:teuthology.orchestra.run.smithi131.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T00:53:48.246 INFO:teuthology.orchestra.run.smithi131.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-23T00:53:48.246 INFO:teuthology.orchestra.run.smithi131.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-23T00:53:48.246 INFO:teuthology.orchestra.run.smithi131.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-23T00:53:48.247 INFO:teuthology.orchestra.run.smithi131.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T00:53:48.248 INFO:teuthology.orchestra.run.smithi131.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T00:53:48.250 INFO:teuthology.orchestra.run.smithi131.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-23T00:53:48.251 INFO:teuthology.orchestra.run.smithi131.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-23T00:53:48.403 INFO:teuthology.orchestra.run.smithi092.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 38 kB/s | 6.4 kB 00:00 2023-06-23T00:53:48.720 INFO:teuthology.orchestra.run.smithi092.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 2.7 MB/s | 873 kB 00:00 2023-06-23T00:53:48.737 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2023-06-23T00:53:48.748 INFO:teuthology.orchestra.run.smithi131.stdout:====================================================================================================================== 2023-06-23T00:53:48.749 INFO:teuthology.orchestra.run.smithi131.stdout: Package Arch Version Repository Size 2023-06-23T00:53:48.749 INFO:teuthology.orchestra.run.smithi131.stdout:====================================================================================================================== 2023-06-23T00:53:48.749 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2023-06-23T00:53:48.749 INFO:teuthology.orchestra.run.smithi131.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-23T00:53:48.749 INFO:teuthology.orchestra.run.smithi131.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T00:53:48.749 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T00:53:48.750 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T00:53:48.750 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T00:53:48.750 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T00:53:48.750 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T00:53:48.750 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T00:53:48.750 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T00:53:48.750 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T00:53:48.750 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T00:53:48.751 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T00:53:48.751 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T00:53:48.751 INFO:teuthology.orchestra.run.smithi131.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T00:53:48.751 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T00:53:48.751 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T00:53:48.751 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-23T00:53:48.752 INFO:teuthology.orchestra.run.smithi131.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-23T00:53:48.752 INFO:teuthology.orchestra.run.smithi131.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-23T00:53:48.752 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T00:53:48.753 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T00:53:48.753 INFO:teuthology.orchestra.run.smithi131.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T00:53:48.753 INFO:teuthology.orchestra.run.smithi131.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-23T00:53:48.753 INFO:teuthology.orchestra.run.smithi131.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-23T00:53:48.753 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-23T00:53:48.753 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T00:53:48.753 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T00:53:48.754 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T00:53:48.754 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T00:53:48.754 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T00:53:48.754 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T00:53:48.754 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T00:53:48.754 INFO:teuthology.orchestra.run.smithi131.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-23T00:53:48.755 INFO:teuthology.orchestra.run.smithi131.stdout:Upgrading: 2023-06-23T00:53:48.755 INFO:teuthology.orchestra.run.smithi131.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T00:53:48.755 INFO:teuthology.orchestra.run.smithi131.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T00:53:48.755 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2023-06-23T00:53:48.755 INFO:teuthology.orchestra.run.smithi131.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-23T00:53:48.755 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T00:53:48.755 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T00:53:48.756 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T00:53:48.756 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T00:53:48.756 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T00:53:48.756 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T00:53:48.756 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T00:53:48.756 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T00:53:48.756 INFO:teuthology.orchestra.run.smithi131.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-23T00:53:48.757 INFO:teuthology.orchestra.run.smithi131.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T00:53:48.757 INFO:teuthology.orchestra.run.smithi131.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-23T00:53:48.757 INFO:teuthology.orchestra.run.smithi131.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-23T00:53:48.757 INFO:teuthology.orchestra.run.smithi131.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-23T00:53:48.757 INFO:teuthology.orchestra.run.smithi131.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T00:53:48.757 INFO:teuthology.orchestra.run.smithi131.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-23T00:53:48.757 INFO:teuthology.orchestra.run.smithi131.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-23T00:53:48.757 INFO:teuthology.orchestra.run.smithi131.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T00:53:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-23T00:53:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-23T00:53:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T00:53:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-23T00:53:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T00:53:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-23T00:53:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-23T00:53:48.758 INFO:teuthology.orchestra.run.smithi131.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T00:53:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-23T00:53:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-23T00:53:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-23T00:53:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-23T00:53:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-23T00:53:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-23T00:53:48.759 INFO:teuthology.orchestra.run.smithi131.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-23T00:53:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-23T00:53:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T00:53:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-23T00:53:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T00:53:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T00:53:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T00:53:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T00:53:48.760 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T00:53:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T00:53:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T00:53:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T00:53:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T00:53:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T00:53:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T00:53:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-23T00:53:48.761 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-23T00:53:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-23T00:53:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T00:53:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T00:53:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T00:53:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-23T00:53:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-23T00:53:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T00:53:48.762 INFO:teuthology.orchestra.run.smithi131.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T00:53:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-23T00:53:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-23T00:53:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-23T00:53:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T00:53:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T00:53:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-23T00:53:48.763 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-23T00:53:48.764 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-23T00:53:48.764 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-23T00:53:48.764 INFO:teuthology.orchestra.run.smithi131.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T00:53:48.764 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T00:53:48.764 INFO:teuthology.orchestra.run.smithi131.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T00:53:48.764 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T00:53:48.764 INFO:teuthology.orchestra.run.smithi131.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-23T00:53:48.765 INFO:teuthology.orchestra.run.smithi131.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T00:53:48.765 INFO:teuthology.orchestra.run.smithi131.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T00:53:48.766 INFO:teuthology.orchestra.run.smithi131.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T00:53:48.766 INFO:teuthology.orchestra.run.smithi131.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T00:53:48.766 INFO:teuthology.orchestra.run.smithi131.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T00:53:48.766 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T00:53:48.766 INFO:teuthology.orchestra.run.smithi131.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T00:53:48.766 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T00:53:48.766 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-23T00:53:48.767 INFO:teuthology.orchestra.run.smithi131.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T00:53:48.767 INFO:teuthology.orchestra.run.smithi131.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-23T00:53:48.767 INFO:teuthology.orchestra.run.smithi131.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T00:53:48.767 INFO:teuthology.orchestra.run.smithi131.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-23T00:53:48.767 INFO:teuthology.orchestra.run.smithi131.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-23T00:53:48.767 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:53:48.767 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2023-06-23T00:53:48.768 INFO:teuthology.orchestra.run.smithi131.stdout:====================================================================================================================== 2023-06-23T00:53:48.768 INFO:teuthology.orchestra.run.smithi131.stdout:Install 118 Packages 2023-06-23T00:53:48.768 INFO:teuthology.orchestra.run.smithi131.stdout:Upgrade 2 Packages 2023-06-23T00:53:48.768 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:53:48.769 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 201 M 2023-06-23T00:53:48.769 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2023-06-23T00:53:48.787 INFO:teuthology.orchestra.run.smithi092.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 2.4 MB/s | 162 kB 00:00 2023-06-23T00:53:49.040 INFO:teuthology.orchestra.run.smithi131.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 26 kB/s | 6.4 kB 00:00 2023-06-23T00:53:49.168 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T00:53:49.168 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T00:53:49.182 INFO:teuthology.orchestra.run.smithi046.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-23T00:53:49.182 INFO:teuthology.orchestra.run.smithi046.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-23T00:53:49.183 INFO:teuthology.orchestra.run.smithi046.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-23T00:53:49.186 INFO:teuthology.orchestra.run.smithi046.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T00:53:49.186 INFO:teuthology.orchestra.run.smithi046.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T00:53:49.187 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-23T00:53:49.188 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-23T00:53:49.415 INFO:teuthology.orchestra.run.smithi092.stdout:(4/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 3.7 MB/s | 2.3 MB 00:00 2023-06-23T00:53:49.532 INFO:teuthology.orchestra.run.smithi092.stdout:(5/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 4.0 MB/s | 5.2 MB 00:01 2023-06-23T00:53:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2023-06-23T00:53:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:====================================================================================================================== 2023-06-23T00:53:49.723 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2023-06-23T00:53:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:====================================================================================================================== 2023-06-23T00:53:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2023-06-23T00:53:49.724 INFO:teuthology.orchestra.run.smithi046.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-23T00:53:49.724 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T00:53:49.724 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T00:53:49.724 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T00:53:49.724 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T00:53:49.724 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T00:53:49.725 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T00:53:49.725 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T00:53:49.725 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T00:53:49.725 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T00:53:49.725 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T00:53:49.725 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T00:53:49.725 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T00:53:49.726 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T00:53:49.726 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T00:53:49.726 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T00:53:49.726 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-23T00:53:49.726 INFO:teuthology.orchestra.run.smithi046.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-23T00:53:49.726 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-23T00:53:49.727 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T00:53:49.727 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T00:53:49.727 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T00:53:49.727 INFO:teuthology.orchestra.run.smithi046.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-23T00:53:49.727 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-23T00:53:49.727 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-23T00:53:49.727 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T00:53:49.728 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T00:53:49.728 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T00:53:49.728 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T00:53:49.729 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T00:53:49.729 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T00:53:49.729 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T00:53:49.729 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-23T00:53:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2023-06-23T00:53:49.729 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T00:53:49.729 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T00:53:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2023-06-23T00:53:49.730 INFO:teuthology.orchestra.run.smithi046.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-23T00:53:49.730 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T00:53:49.730 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T00:53:49.730 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T00:53:49.731 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T00:53:49.731 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T00:53:49.731 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T00:53:49.732 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T00:53:49.732 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T00:53:49.732 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-23T00:53:49.733 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T00:53:49.733 INFO:teuthology.orchestra.run.smithi046.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-23T00:53:49.733 INFO:teuthology.orchestra.run.smithi046.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-23T00:53:49.733 INFO:teuthology.orchestra.run.smithi046.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-23T00:53:49.733 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T00:53:49.734 INFO:teuthology.orchestra.run.smithi046.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-23T00:53:49.734 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-23T00:53:49.734 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T00:53:49.734 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-23T00:53:49.734 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-23T00:53:49.734 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T00:53:49.734 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-23T00:53:49.734 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T00:53:49.735 INFO:teuthology.orchestra.run.smithi046.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-23T00:53:49.735 INFO:teuthology.orchestra.run.smithi046.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-23T00:53:49.735 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T00:53:49.735 INFO:teuthology.orchestra.run.smithi046.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-23T00:53:49.735 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-23T00:53:49.735 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-23T00:53:49.735 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-23T00:53:49.736 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-23T00:53:49.736 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-23T00:53:49.736 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-23T00:53:49.736 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-23T00:53:49.736 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T00:53:49.736 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-23T00:53:49.737 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T00:53:49.737 INFO:teuthology.orchestra.run.smithi046.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T00:53:49.737 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T00:53:49.737 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T00:53:49.737 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T00:53:49.738 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T00:53:49.738 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T00:53:49.738 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T00:53:49.738 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T00:53:49.738 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T00:53:49.738 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T00:53:49.738 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-23T00:53:49.738 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-23T00:53:49.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-23T00:53:49.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T00:53:49.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T00:53:49.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T00:53:49.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-23T00:53:49.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-23T00:53:49.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T00:53:49.739 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T00:53:49.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-23T00:53:49.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-23T00:53:49.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-23T00:53:49.740 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T00:53:49.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T00:53:49.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-23T00:53:49.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-23T00:53:49.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-23T00:53:49.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-23T00:53:49.741 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T00:53:49.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T00:53:49.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T00:53:49.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T00:53:49.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-23T00:53:49.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T00:53:49.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T00:53:49.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T00:53:49.742 INFO:teuthology.orchestra.run.smithi046.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T00:53:49.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T00:53:49.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T00:53:49.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T00:53:49.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T00:53:49.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-23T00:53:49.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T00:53:49.743 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-23T00:53:49.743 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T00:53:49.744 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-23T00:53:49.744 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-23T00:53:49.744 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:53:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2023-06-23T00:53:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:====================================================================================================================== 2023-06-23T00:53:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:Install 118 Packages 2023-06-23T00:53:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 2 Packages 2023-06-23T00:53:49.745 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:53:49.745 INFO:teuthology.orchestra.run.smithi092.stdout:(6/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 4.8 MB/s | 1.6 MB 00:00 2023-06-23T00:53:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 201 M 2023-06-23T00:53:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2023-06-23T00:53:49.881 INFO:teuthology.orchestra.run.smithi131.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 1.0 MB/s | 873 kB 00:00 2023-06-23T00:53:50.024 INFO:teuthology.orchestra.run.smithi131.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 1.1 MB/s | 162 kB 00:00 2023-06-23T00:53:50.156 INFO:teuthology.orchestra.run.smithi046.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 23 kB/s | 6.4 kB 00:00 2023-06-23T00:53:50.279 INFO:teuthology.orchestra.run.smithi092.stdout:(7/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 5.3 MB/s | 3.9 MB 00:00 2023-06-23T00:53:50.840 INFO:teuthology.orchestra.run.smithi092.stdout:(8/120): ceph-common-17.2.6-719.g42ddc46e.el8.x 9.0 MB/s | 23 MB 00:02 2023-06-23T00:53:50.842 INFO:teuthology.orchestra.run.smithi046.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 1.2 MB/s | 873 kB 00:00 2023-06-23T00:53:50.903 INFO:teuthology.orchestra.run.smithi092.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 384 kB/s | 24 kB 00:00 2023-06-23T00:53:50.982 INFO:teuthology.orchestra.run.smithi046.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 1.1 MB/s | 162 kB 00:00 2023-06-23T00:53:50.992 INFO:teuthology.orchestra.run.smithi131.stdout:(4/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 2.4 MB/s | 2.3 MB 00:00 2023-06-23T00:53:51.671 INFO:teuthology.orchestra.run.smithi131.stdout:(5/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 2.3 MB/s | 1.6 MB 00:00 2023-06-23T00:53:51.834 INFO:teuthology.orchestra.run.smithi046.stdout:(4/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 2.6 MB/s | 5.2 MB 00:01 2023-06-23T00:53:52.345 INFO:teuthology.orchestra.run.smithi046.stdout:(5/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 3.1 MB/s | 1.6 MB 00:00 2023-06-23T00:53:52.700 INFO:teuthology.orchestra.run.smithi131.stdout:(6/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 1.3 MB/s | 5.2 MB 00:03 2023-06-23T00:53:52.738 INFO:teuthology.orchestra.run.smithi046.stdout:(6/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 1.3 MB/s | 2.3 MB 00:01 2023-06-23T00:53:53.137 INFO:teuthology.orchestra.run.smithi131.stdout:(7/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 2.7 MB/s | 3.9 MB 00:01 2023-06-23T00:53:53.389 INFO:teuthology.orchestra.run.smithi046.stdout:(7/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 3.8 MB/s | 3.9 MB 00:01 2023-06-23T00:53:53.436 INFO:teuthology.orchestra.run.smithi092.stdout:(10/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 4.5 MB/s | 17 MB 00:03 2023-06-23T00:53:53.532 INFO:teuthology.orchestra.run.smithi092.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 2.6 MB/s | 255 kB 00:00 2023-06-23T00:53:53.617 INFO:teuthology.orchestra.run.smithi092.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 295 kB/s | 25 kB 00:00 2023-06-23T00:53:53.843 INFO:teuthology.orchestra.run.smithi092.stdout:(13/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 3.4 MB/s | 794 kB 00:00 2023-06-23T00:53:53.927 INFO:teuthology.orchestra.run.smithi092.stdout:(14/120): libcephsqlite-17.2.6-719.g42ddc46e.el 2.0 MB/s | 174 kB 00:00 2023-06-23T00:53:54.083 INFO:teuthology.orchestra.run.smithi092.stdout:(15/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 4.4 MB/s | 17 MB 00:03 2023-06-23T00:53:54.085 INFO:teuthology.orchestra.run.smithi092.stdout:(16/120): librados-devel-17.2.6-719.g42ddc46e.e 825 kB/s | 130 kB 00:00 2023-06-23T00:53:54.241 INFO:teuthology.orchestra.run.smithi092.stdout:(17/120): libradosstriper1-17.2.6-719.g42ddc46e 3.2 MB/s | 506 kB 00:00 2023-06-23T00:53:54.313 INFO:teuthology.orchestra.run.smithi092.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 579 kB/s | 45 kB 00:00 2023-06-23T00:53:54.395 INFO:teuthology.orchestra.run.smithi092.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 1.4 MB/s | 115 kB 00:00 2023-06-23T00:53:54.481 INFO:teuthology.orchestra.run.smithi092.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 2.2 MB/s | 190 kB 00:00 2023-06-23T00:53:54.648 INFO:teuthology.orchestra.run.smithi092.stdout:(21/120): python3-rados-17.2.6-719.g42ddc46e.el 2.2 MB/s | 371 kB 00:00 2023-06-23T00:53:54.820 INFO:teuthology.orchestra.run.smithi092.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 2.0 MB/s | 357 kB 00:00 2023-06-23T00:53:54.909 INFO:teuthology.orchestra.run.smithi092.stdout:(23/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 1.2 MB/s | 107 kB 00:00 2023-06-23T00:53:54.993 INFO:teuthology.orchestra.run.smithi092.stdout:(24/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.0 MB/s | 91 kB 00:00 2023-06-23T00:53:55.305 INFO:teuthology.orchestra.run.smithi092.stdout:(25/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 4.5 MB/s | 5.5 MB 00:01 2023-06-23T00:53:55.390 INFO:teuthology.orchestra.run.smithi092.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 2.0 MB/s | 173 kB 00:00 2023-06-23T00:53:55.470 INFO:teuthology.orchestra.run.smithi092.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 290 kB/s | 23 kB 00:00 2023-06-23T00:53:55.554 INFO:teuthology.orchestra.run.smithi092.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 1.4 MB/s | 121 kB 00:00 2023-06-23T00:53:55.699 INFO:teuthology.orchestra.run.smithi046.stdout:(8/120): ceph-common-17.2.6-719.g42ddc46e.el8.x 4.0 MB/s | 23 MB 00:05 2023-06-23T00:53:55.775 INFO:teuthology.orchestra.run.smithi046.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 322 kB/s | 24 kB 00:00 2023-06-23T00:53:56.001 INFO:teuthology.orchestra.run.smithi092.stdout:(29/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 3.4 MB/s | 3.5 MB 00:01 2023-06-23T00:53:56.005 INFO:teuthology.orchestra.run.smithi092.stdout:(30/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 3.8 MB/s | 1.7 MB 00:00 2023-06-23T00:53:56.076 INFO:teuthology.orchestra.run.smithi092.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 3.3 MB/s | 238 kB 00:00 2023-06-23T00:53:56.142 INFO:teuthology.orchestra.run.smithi092.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 715 kB/s | 47 kB 00:00 2023-06-23T00:53:56.208 INFO:teuthology.orchestra.run.smithi092.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 219 kB/s | 14 kB 00:00 2023-06-23T00:53:56.284 INFO:teuthology.orchestra.run.smithi092.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 3.4 MB/s | 264 kB 00:00 2023-06-23T00:53:56.355 INFO:teuthology.orchestra.run.smithi092.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.1 MB/s | 80 kB 00:00 2023-06-23T00:53:56.424 INFO:teuthology.orchestra.run.smithi092.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 262 kB/s | 18 kB 00:00 2023-06-23T00:53:56.440 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:53:56.443 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:53:56.866 INFO:teuthology.orchestra.run.smithi092.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 694 kB/s | 306 kB 00:00 2023-06-23T00:53:56.866 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:53:56.867 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:53:56.940 INFO:teuthology.orchestra.run.smithi092.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 801 kB/s | 59 kB 00:00 2023-06-23T00:53:56.940 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:53:56.940 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:53:57.011 INFO:teuthology.orchestra.run.smithi092.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 1.0 MB/s | 75 kB 00:00 2023-06-23T00:53:57.157 INFO:teuthology.orchestra.run.smithi092.stdout:(40/120): python3-asyncssh-2.7.0-2.el8.noarch.r 3.3 MB/s | 476 kB 00:00 2023-06-23T00:53:57.226 INFO:teuthology.orchestra.run.smithi092.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 618 kB/s | 44 kB 00:00 2023-06-23T00:53:57.297 INFO:teuthology.orchestra.run.smithi092.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 2.5 MB/s | 185 kB 00:00 2023-06-23T00:53:57.368 INFO:teuthology.orchestra.run.smithi092.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 467 kB/s | 33 kB 00:00 2023-06-23T00:53:57.439 INFO:teuthology.orchestra.run.smithi092.stdout:(44/120): python3-certifi-2018.10.15-7.el8.noar 232 kB/s | 16 kB 00:00 2023-06-23T00:53:57.511 INFO:teuthology.orchestra.run.smithi092.stdout:(45/120): python3-cheroot-8.5.2-1.el8.noarch.rp 2.3 MB/s | 173 kB 00:00 2023-06-23T00:53:57.586 INFO:teuthology.orchestra.run.smithi092.stdout:(46/120): python3-cherrypy-18.4.0-1.el8.noarch. 5.0 MB/s | 384 kB 00:00 2023-06-23T00:53:57.657 INFO:teuthology.orchestra.run.smithi092.stdout:(47/120): python3-google-auth-1.1.1-10.el8.noar 1.1 MB/s | 81 kB 00:00 2023-06-23T00:53:57.729 INFO:teuthology.orchestra.run.smithi092.stdout:(48/120): python3-jaraco-6.2-6.el8.noarch.rpm 158 kB/s | 11 kB 00:00 2023-06-23T00:53:57.798 INFO:teuthology.orchestra.run.smithi092.stdout:(49/120): python3-jaraco-functools-2.0-4.el8.no 250 kB/s | 18 kB 00:00 2023-06-23T00:53:57.946 INFO:teuthology.orchestra.run.smithi092.stdout:(50/120): python3-kubernetes-11.0.0-6.el8.noarc 7.9 MB/s | 1.2 MB 00:00 2023-06-23T00:53:57.968 INFO:teuthology.orchestra.run.smithi131.stdout:(8/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8. 3.5 MB/s | 17 MB 00:04 2023-06-23T00:53:58.016 INFO:teuthology.orchestra.run.smithi092.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 690 kB/s | 49 kB 00:00 2023-06-23T00:53:58.022 INFO:teuthology.orchestra.run.smithi131.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 454 kB/s | 24 kB 00:00 2023-06-23T00:53:58.087 INFO:teuthology.orchestra.run.smithi092.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 828 kB/s | 59 kB 00:00 2023-06-23T00:53:58.160 INFO:teuthology.orchestra.run.smithi092.stdout:(53/120): python3-natsort-7.1.1-2.el8.noarch.rp 850 kB/s | 60 kB 00:00 2023-06-23T00:53:58.248 INFO:teuthology.orchestra.run.smithi092.stdout:(54/120): python3-pecan-1.3.2-9.el8.noarch.rpm 3.1 MB/s | 283 kB 00:00 2023-06-23T00:53:58.318 INFO:teuthology.orchestra.run.smithi092.stdout:(55/120): python3-portend-2.6-1.el8.noarch.rpm 221 kB/s | 16 kB 00:00 2023-06-23T00:53:58.368 INFO:teuthology.orchestra.run.smithi046.stdout:(10/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 3.4 MB/s | 17 MB 00:04 2023-06-23T00:53:58.390 INFO:teuthology.orchestra.run.smithi092.stdout:(56/120): python3-repoze-lru-0.7-6.el8.noarch.r 475 kB/s | 34 kB 00:00 2023-06-23T00:53:58.462 INFO:teuthology.orchestra.run.smithi092.stdout:(57/120): python3-routes-2.4.1-12.el8.noarch.rp 2.6 MB/s | 196 kB 00:00 2023-06-23T00:53:58.490 INFO:teuthology.orchestra.run.smithi046.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 2.0 MB/s | 255 kB 00:00 2023-06-23T00:53:58.534 INFO:teuthology.orchestra.run.smithi092.stdout:(58/120): python3-rsa-4.8-1.el8.noarch.rpm 860 kB/s | 61 kB 00:00 2023-06-23T00:53:58.552 INFO:teuthology.orchestra.run.smithi046.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 420 kB/s | 25 kB 00:00 2023-06-23T00:53:58.605 INFO:teuthology.orchestra.run.smithi092.stdout:(59/120): python3-simplegeneric-0.8.1-17.el8.no 266 kB/s | 19 kB 00:00 2023-06-23T00:53:58.675 INFO:teuthology.orchestra.run.smithi092.stdout:(60/120): python3-singledispatch-3.4.0.3-18.el8 339 kB/s | 24 kB 00:00 2023-06-23T00:53:58.746 INFO:teuthology.orchestra.run.smithi092.stdout:(61/120): python3-tempora-1.14.1-5.el8.noarch.r 415 kB/s | 29 kB 00:00 2023-06-23T00:53:58.801 INFO:teuthology.orchestra.run.smithi046.stdout:(13/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 3.1 MB/s | 794 kB 00:00 2023-06-23T00:53:58.815 INFO:teuthology.orchestra.run.smithi092.stdout:(62/120): python3-trustme-0.6.0-4.el8.noarch.rp 391 kB/s | 27 kB 00:00 2023-06-23T00:53:58.864 INFO:teuthology.orchestra.run.smithi046.stdout:(14/120): libcephsqlite-17.2.6-719.g42ddc46e.el 2.8 MB/s | 174 kB 00:00 2023-06-23T00:53:58.889 INFO:teuthology.orchestra.run.smithi092.stdout:(63/120): python3-waitress-1.4.3-1.el8.noarch.r 3.4 MB/s | 255 kB 00:00 2023-06-23T00:53:58.926 INFO:teuthology.orchestra.run.smithi046.stdout:(15/120): librados-devel-17.2.6-719.g42ddc46e.e 2.0 MB/s | 130 kB 00:00 2023-06-23T00:53:58.964 INFO:teuthology.orchestra.run.smithi092.stdout:(64/120): python3-webob-1.8.5-1.el8.1.noarch.rp 3.3 MB/s | 251 kB 00:00 2023-06-23T00:53:59.046 INFO:teuthology.orchestra.run.smithi092.stdout:(65/120): ceph-mgr-diskprediction-local-17.2.6- 2.4 MB/s | 7.4 MB 00:03 2023-06-23T00:53:59.046 INFO:teuthology.orchestra.run.smithi092.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 738 kB/s | 61 kB 00:00 2023-06-23T00:53:59.096 INFO:teuthology.orchestra.run.smithi046.stdout:(16/120): libradosstriper1-17.2.6-719.g42ddc46e 2.9 MB/s | 506 kB 00:00 2023-06-23T00:53:59.117 INFO:teuthology.orchestra.run.smithi092.stdout:(67/120): python3-zc-lockfile-2.0-2.el8.noarch. 326 kB/s | 23 kB 00:00 2023-06-23T00:53:59.209 INFO:teuthology.orchestra.run.smithi092.stdout:(68/120): thrift-0.13.0-2.el8.x86_64.rpm 19 MB/s | 1.7 MB 00:00 2023-06-23T00:53:59.213 INFO:teuthology.orchestra.run.smithi092.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 18 MB/s | 70 kB 00:00 2023-06-23T00:53:59.466 INFO:teuthology.orchestra.run.smithi092.stdout:(70/120): python3-scipy-1.0.0-20.module+el8.1.0 57 MB/s | 15 MB 00:00 2023-06-23T00:53:59.468 INFO:teuthology.orchestra.run.smithi092.stdout:(71/120): python3-webtest-2.0.33-1.el8.noarch.r 202 kB/s | 85 kB 00:00 2023-06-23T00:53:59.485 INFO:teuthology.orchestra.run.smithi092.stdout:(72/120): python3-prettytable-0.7.2-14.el8.noar 2.4 MB/s | 44 kB 00:00 2023-06-23T00:53:59.498 INFO:teuthology.orchestra.run.smithi092.stdout:(73/120): python3-jsonpatch-1.21-2.el8.noarch.r 886 kB/s | 27 kB 00:00 2023-06-23T00:53:59.500 INFO:teuthology.orchestra.run.smithi092.stdout:(74/120): jq-1.5-12.el8.x86_64.rpm 11 MB/s | 161 kB 00:00 2023-06-23T00:53:59.514 INFO:teuthology.orchestra.run.smithi092.stdout:(75/120): flex-2.6.1-9.el8.x86_64.rpm 19 MB/s | 320 kB 00:00 2023-06-23T00:53:59.520 INFO:teuthology.orchestra.run.smithi092.stdout:(76/120): python3-werkzeug-0.12.2-4.el8.noarch. 22 MB/s | 457 kB 00:00 2023-06-23T00:53:59.540 INFO:teuthology.orchestra.run.smithi092.stdout:(77/120): python3-markupsafe-0.23-19.el8.x86_64 2.0 MB/s | 39 kB 00:00 2023-06-23T00:53:59.562 INFO:teuthology.orchestra.run.smithi092.stdout:(78/120): bison-3.0.4-10.el8.x86_64.rpm 31 MB/s | 688 kB 00:00 2023-06-23T00:53:59.585 INFO:teuthology.orchestra.run.smithi092.stdout:(79/120): python3-babel-2.5.1-5.el8.noarch.rpm 68 MB/s | 4.8 MB 00:00 2023-06-23T00:53:59.588 INFO:teuthology.orchestra.run.smithi092.stdout:(80/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 9.6 MB/s | 259 kB 00:00 2023-06-23T00:53:59.600 INFO:teuthology.orchestra.run.smithi092.stdout:(81/120): python3-numpy-f2py-1.14.3-9.el8.x86_6 15 MB/s | 225 kB 00:00 2023-06-23T00:53:59.603 INFO:teuthology.orchestra.run.smithi092.stdout:(82/120): python3-pyasn1-0.3.7-6.el8.noarch.rpm 8.5 MB/s | 126 kB 00:00 2023-06-23T00:53:59.618 INFO:teuthology.orchestra.run.smithi092.stdout:(83/120): oniguruma-6.8.2-2.el8.x86_64.rpm 13 MB/s | 187 kB 00:00 2023-06-23T00:53:59.639 INFO:teuthology.orchestra.run.smithi092.stdout:(84/120): python3-jinja2-2.10.1-2.el8_0.noarch. 26 MB/s | 538 kB 00:00 2023-06-23T00:53:59.658 INFO:teuthology.orchestra.run.smithi092.stdout:(85/120): python3-numpy-1.14.3-9.el8.x86_64.rpm 65 MB/s | 3.7 MB 00:00 2023-06-23T00:53:59.662 INFO:teuthology.orchestra.run.smithi092.stdout:(86/120): socat-1.7.3.3-2.el8.x86_64.rpm 13 MB/s | 302 kB 00:00 2023-06-23T00:53:59.674 INFO:teuthology.orchestra.run.smithi092.stdout:(87/120): python3-mako-1.0.6-13.el8.noarch.rpm 10 MB/s | 157 kB 00:00 2023-06-23T00:53:59.677 INFO:teuthology.orchestra.run.smithi092.stdout:(88/120): python3-pyasn1-modules-0.3.7-6.el8.no 7.3 MB/s | 110 kB 00:00 2023-06-23T00:53:59.690 INFO:teuthology.orchestra.run.smithi092.stdout:(89/120): python3-jsonpointer-1.10-11.el8.noarc 1.6 MB/s | 20 kB 00:00 2023-06-23T00:53:59.691 INFO:teuthology.orchestra.run.smithi092.stdout:(90/120): librdkafka-0.11.4-1.el8.x86_64.rpm 21 MB/s | 353 kB 00:00 2023-06-23T00:53:59.703 INFO:teuthology.orchestra.run.smithi092.stdout:(91/120): python3-pytz-2017.2-9.el8.noarch.rpm 4.3 MB/s | 54 kB 00:00 2023-06-23T00:53:59.708 INFO:teuthology.orchestra.run.smithi092.stdout:(92/120): libtool-2.4.6-25.el8.x86_64.rpm 35 MB/s | 709 kB 00:00 2023-06-23T00:53:59.779 INFO:teuthology.orchestra.run.smithi092.stdout:(93/120): openblas-threads-0.3.12-1.el8.x86_64. 64 MB/s | 4.7 MB 00:00 2023-06-23T00:53:59.794 INFO:teuthology.orchestra.run.smithi092.stdout:(94/120): openblas-0.3.12-1.el8.x86_64.rpm 54 MB/s | 4.6 MB 00:00 2023-06-23T00:53:59.801 INFO:teuthology.orchestra.run.smithi092.stdout:(95/120): automake-1.16.1-7.el8.noarch.rpm 29 MB/s | 713 kB 00:00 2023-06-23T00:53:59.806 INFO:teuthology.orchestra.run.smithi092.stdout:(96/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 1.5 MB/s | 19 kB 00:00 2023-06-23T00:53:59.816 INFO:teuthology.orchestra.run.smithi092.stdout:(97/120): python3-jwt-1.6.1-2.el8.noarch.rpm 2.9 MB/s | 43 kB 00:00 2023-06-23T00:53:59.821 INFO:teuthology.orchestra.run.smithi092.stdout:(98/120): python3-oauthlib-2.1.0-1.el8.noarch.r 10 MB/s | 155 kB 00:00 2023-06-23T00:53:59.831 INFO:teuthology.orchestra.run.smithi092.stdout:(99/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 14 MB/s | 220 kB 00:00 2023-06-23T00:53:59.835 INFO:teuthology.orchestra.run.smithi092.stdout:(100/120): keyutils-libs-devel-1.5.10-6.el8.x86 3.4 MB/s | 48 kB 00:00 2023-06-23T00:53:59.846 INFO:teuthology.orchestra.run.smithi092.stdout:(101/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 15 MB/s | 229 kB 00:00 2023-06-23T00:53:59.847 INFO:teuthology.orchestra.run.smithi092.stdout:(102/120): python3-requests-oauthlib-1.0.0-1.el 3.4 MB/s | 43 kB 00:00 2023-06-23T00:53:59.860 INFO:teuthology.orchestra.run.smithi092.stdout:(103/120): libverto-devel-0.3.0-5.el8.x86_64.rp 1.5 MB/s | 18 kB 00:00 2023-06-23T00:53:59.879 INFO:teuthology.orchestra.run.smithi092.stdout:(104/120): pcre2-devel-10.32-2.el8.x86_64.rpm 31 MB/s | 605 kB 00:00 2023-06-23T00:53:59.887 INFO:teuthology.orchestra.run.smithi092.stdout:(105/120): openssl-devel-1.1.1g-15.el8_3.x86_64 57 MB/s | 2.3 MB 00:00 2023-06-23T00:53:59.893 INFO:teuthology.orchestra.run.smithi092.stdout:(106/120): fuse-2.9.7-12.el8.x86_64.rpm 6.4 MB/s | 83 kB 00:00 2023-06-23T00:53:59.900 INFO:teuthology.orchestra.run.smithi092.stdout:(107/120): elfutils-libelf-devel-0.182-3.el8.x8 4.5 MB/s | 59 kB 00:00 2023-06-23T00:53:59.906 INFO:teuthology.orchestra.run.smithi092.stdout:(108/120): libquadmath-8.4.1-1.el8.x86_64.rpm 12 MB/s | 169 kB 00:00 2023-06-23T00:53:59.914 INFO:teuthology.orchestra.run.smithi092.stdout:(109/120): libcom_err-devel-1.45.6-1.el8.x86_64 2.7 MB/s | 38 kB 00:00 2023-06-23T00:53:59.923 INFO:teuthology.orchestra.run.smithi092.stdout:(110/120): libsepol-devel-2.9-2.el8.x86_64.rpm 4.9 MB/s | 86 kB 00:00 2023-06-23T00:53:59.929 INFO:teuthology.orchestra.run.smithi092.stdout:(111/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 4.1 MB/s | 58 kB 00:00 2023-06-23T00:53:59.937 INFO:teuthology.orchestra.run.smithi092.stdout:(112/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.4 MB/s | 47 kB 00:00 2023-06-23T00:53:59.946 INFO:teuthology.orchestra.run.smithi092.stdout:(113/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 30 MB/s | 559 kB 00:00 2023-06-23T00:53:59.955 INFO:teuthology.orchestra.run.smithi092.stdout:(114/120): libgfortran-8.4.1-1.el8.x86_64.rpm 34 MB/s | 639 kB 00:00 2023-06-23T00:53:59.960 INFO:teuthology.orchestra.run.smithi092.stdout:(115/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 5.0 MB/s | 70 kB 00:00 2023-06-23T00:53:59.972 INFO:teuthology.orchestra.run.smithi092.stdout:(116/120): libselinux-devel-2.9-5.el8.x86_64.rp 12 MB/s | 200 kB 00:00 2023-06-23T00:53:59.974 INFO:teuthology.orchestra.run.smithi092.stdout:(117/120): libuuid-devel-2.32.1-27.el8.x86_64.r 7.0 MB/s | 97 kB 00:00 2023-06-23T00:54:00.168 INFO:teuthology.orchestra.run.smithi046.stdout:(17/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 2.2 MB/s | 17 MB 00:07 2023-06-23T00:54:00.224 INFO:teuthology.orchestra.run.smithi046.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 812 kB/s | 45 kB 00:00 2023-06-23T00:54:00.293 INFO:teuthology.orchestra.run.smithi046.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 1.6 MB/s | 115 kB 00:00 2023-06-23T00:54:00.372 INFO:teuthology.orchestra.run.smithi046.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 2.4 MB/s | 190 kB 00:00 2023-06-23T00:54:00.526 INFO:teuthology.orchestra.run.smithi046.stdout:(21/120): python3-rados-17.2.6-719.g42ddc46e.el 2.3 MB/s | 371 kB 00:00 2023-06-23T00:54:00.660 INFO:teuthology.orchestra.run.smithi046.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 2.6 MB/s | 357 kB 00:00 2023-06-23T00:54:00.700 INFO:teuthology.orchestra.run.smithi046.stdout:(23/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 3.5 MB/s | 5.5 MB 00:01 2023-06-23T00:54:00.719 INFO:teuthology.orchestra.run.smithi046.stdout:(24/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 1.7 MB/s | 107 kB 00:00 2023-06-23T00:54:00.754 INFO:teuthology.orchestra.run.smithi046.stdout:(25/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.8 MB/s | 91 kB 00:00 2023-06-23T00:54:00.809 INFO:teuthology.orchestra.run.smithi046.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 2.9 MB/s | 173 kB 00:00 2023-06-23T00:54:00.867 INFO:teuthology.orchestra.run.smithi046.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 397 kB/s | 23 kB 00:00 2023-06-23T00:54:00.928 INFO:teuthology.orchestra.run.smithi046.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 2.0 MB/s | 121 kB 00:00 2023-06-23T00:54:00.953 INFO:teuthology.orchestra.run.smithi092.stdout:(118/120): ceph-test-17.2.6-719.g42ddc46e.el8.x 5.1 MB/s | 52 MB 00:10 2023-06-23T00:54:01.271 INFO:teuthology.orchestra.run.smithi092.stdout:(119/120): librados2-17.2.6-719.g42ddc46e.el8.x 2.7 MB/s | 3.5 MB 00:01 2023-06-23T00:54:01.417 INFO:teuthology.orchestra.run.smithi046.stdout:(29/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 3.4 MB/s | 1.7 MB 00:00 2023-06-23T00:54:01.434 INFO:teuthology.orchestra.run.smithi092.stdout:(120/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 2.5 MB/s | 3.6 MB 00:01 2023-06-23T00:54:01.434 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-06-23T00:54:01.434 INFO:teuthology.orchestra.run.smithi092.stdout:Total 15 MB/s | 201 MB 00:13 2023-06-23T00:54:01.482 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-06-23T00:54:01.594 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-06-23T00:54:01.594 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-06-23T00:54:01.638 INFO:teuthology.orchestra.run.smithi046.stdout:(30/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 3.8 MB/s | 3.5 MB 00:00 2023-06-23T00:54:01.691 INFO:teuthology.orchestra.run.smithi046.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 4.6 MB/s | 238 kB 00:00 2023-06-23T00:54:01.736 INFO:teuthology.orchestra.run.smithi046.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 1.0 MB/s | 47 kB 00:00 2023-06-23T00:54:01.777 INFO:teuthology.orchestra.run.smithi046.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 349 kB/s | 14 kB 00:00 2023-06-23T00:54:02.037 INFO:teuthology.orchestra.run.smithi131.stdout:(10/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 1.8 MB/s | 17 MB 00:09 2023-06-23T00:54:02.081 INFO:teuthology.orchestra.run.smithi131.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 5.7 MB/s | 255 kB 00:00 2023-06-23T00:54:02.102 INFO:teuthology.orchestra.run.smithi131.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 1.1 MB/s | 25 kB 00:00 2023-06-23T00:54:02.231 INFO:teuthology.orchestra.run.smithi131.stdout:(13/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 6.0 MB/s | 794 kB 00:00 2023-06-23T00:54:02.239 INFO:teuthology.orchestra.run.smithi046.stdout:(34/120): ceph-test-17.2.6-719.g42ddc46e.el8.x8 8.0 MB/s | 52 MB 00:06 2023-06-23T00:54:02.240 INFO:teuthology.orchestra.run.smithi046.stdout:(35/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 569 kB/s | 264 kB 00:00 2023-06-23T00:54:02.264 INFO:teuthology.orchestra.run.smithi046.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 762 kB/s | 18 kB 00:00 2023-06-23T00:54:02.278 INFO:teuthology.orchestra.run.smithi131.stdout:(14/120): libcephsqlite-17.2.6-719.g42ddc46e.el 4.0 MB/s | 174 kB 00:00 2023-06-23T00:54:02.282 INFO:teuthology.orchestra.run.smithi046.stdout:(37/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.9 MB/s | 80 kB 00:00 2023-06-23T00:54:02.282 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:02.282 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:02.285 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:02.285 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:02.304 INFO:teuthology.orchestra.run.smithi131.stdout:(15/120): librados-devel-17.2.6-719.g42ddc46e.e 4.9 MB/s | 130 kB 00:00 2023-06-23T00:54:02.405 INFO:teuthology.orchestra.run.smithi131.stdout:(16/120): libradosstriper1-17.2.6-719.g42ddc46e 4.7 MB/s | 506 kB 00:00 2023-06-23T00:54:02.657 INFO:teuthology.orchestra.run.smithi046.stdout:(38/120): ceph-mgr-diskprediction-local-17.2.6- 6.0 MB/s | 7.4 MB 00:01 2023-06-23T00:54:02.694 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:02.696 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:03.015 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-06-23T00:54:03.015 INFO:teuthology.orchestra.run.smithi131.stdout:(17/120): ceph-common-17.2.6-719.g42ddc46e.el8. 1.6 MB/s | 23 MB 00:14 2023-06-23T00:54:03.018 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-06-23T00:54:03.041 INFO:teuthology.orchestra.run.smithi131.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 1.6 MB/s | 45 kB 00:00 2023-06-23T00:54:03.064 INFO:teuthology.orchestra.run.smithi131.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 4.9 MB/s | 115 kB 00:00 2023-06-23T00:54:03.106 INFO:teuthology.orchestra.run.smithi131.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 4.4 MB/s | 190 kB 00:00 2023-06-23T00:54:03.169 INFO:teuthology.orchestra.run.smithi131.stdout:(21/120): python3-rados-17.2.6-719.g42ddc46e.el 5.8 MB/s | 371 kB 00:00 2023-06-23T00:54:03.232 INFO:teuthology.orchestra.run.smithi131.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 5.5 MB/s | 357 kB 00:00 2023-06-23T00:54:03.254 INFO:teuthology.orchestra.run.smithi131.stdout:(23/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 4.8 MB/s | 107 kB 00:00 2023-06-23T00:54:03.277 INFO:teuthology.orchestra.run.smithi131.stdout:(24/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 6.4 MB/s | 5.5 MB 00:00 2023-06-23T00:54:03.278 INFO:teuthology.orchestra.run.smithi131.stdout:(25/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 3.7 MB/s | 91 kB 00:00 2023-06-23T00:54:03.321 INFO:teuthology.orchestra.run.smithi131.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 4.0 MB/s | 173 kB 00:00 2023-06-23T00:54:03.342 INFO:teuthology.orchestra.run.smithi131.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 1.1 MB/s | 23 kB 00:00 2023-06-23T00:54:03.363 INFO:teuthology.orchestra.run.smithi131.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 5.6 MB/s | 121 kB 00:00 2023-06-23T00:54:03.602 INFO:teuthology.orchestra.run.smithi131.stdout:(29/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 7.1 MB/s | 1.7 MB 00:00 2023-06-23T00:54:03.687 INFO:teuthology.orchestra.run.smithi131.stdout:(30/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 8.5 MB/s | 3.5 MB 00:00 2023-06-23T00:54:03.731 INFO:teuthology.orchestra.run.smithi131.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 5.3 MB/s | 238 kB 00:00 2023-06-23T00:54:03.753 INFO:teuthology.orchestra.run.smithi131.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 2.2 MB/s | 47 kB 00:00 2023-06-23T00:54:03.774 INFO:teuthology.orchestra.run.smithi131.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 693 kB/s | 14 kB 00:00 2023-06-23T00:54:03.816 INFO:teuthology.orchestra.run.smithi131.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 6.1 MB/s | 264 kB 00:00 2023-06-23T00:54:03.838 INFO:teuthology.orchestra.run.smithi131.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 3.7 MB/s | 80 kB 00:00 2023-06-23T00:54:03.859 INFO:teuthology.orchestra.run.smithi131.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 854 kB/s | 18 kB 00:00 2023-06-23T00:54:03.874 INFO:teuthology.orchestra.run.smithi131.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:03.877 INFO:teuthology.orchestra.run.smithi131.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:03.984 INFO:teuthology.orchestra.run.smithi131.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 2.4 MB/s | 306 kB 00:00 2023-06-23T00:54:03.985 INFO:teuthology.orchestra.run.smithi131.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:03.985 INFO:teuthology.orchestra.run.smithi131.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:04.004 INFO:teuthology.orchestra.run.smithi131.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 2.8 MB/s | 59 kB 00:00 2023-06-23T00:54:04.005 INFO:teuthology.orchestra.run.smithi131.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:04.006 INFO:teuthology.orchestra.run.smithi131.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:54:04.025 INFO:teuthology.orchestra.run.smithi131.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 3.6 MB/s | 75 kB 00:00 2023-06-23T00:54:04.063 INFO:teuthology.orchestra.run.smithi131.stdout:(40/120): python3-asyncssh-2.7.0-2.el8.noarch.r 12 MB/s | 476 kB 00:00 2023-06-23T00:54:04.082 INFO:teuthology.orchestra.run.smithi131.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 2.3 MB/s | 44 kB 00:00 2023-06-23T00:54:04.101 INFO:teuthology.orchestra.run.smithi131.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 9.2 MB/s | 185 kB 00:00 2023-06-23T00:54:04.121 INFO:teuthology.orchestra.run.smithi131.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 1.8 MB/s | 33 kB 00:00 2023-06-23T00:54:04.137 INFO:teuthology.orchestra.run.smithi131.stdout:(44/120): python3-certifi-2018.10.15-7.el8.noar 932 kB/s | 16 kB 00:00 2023-06-23T00:54:04.157 INFO:teuthology.orchestra.run.smithi131.stdout:(45/120): python3-cheroot-8.5.2-1.el8.noarch.rp 8.7 MB/s | 173 kB 00:00 2023-06-23T00:54:04.179 INFO:teuthology.orchestra.run.smithi131.stdout:(46/120): python3-cherrypy-18.4.0-1.el8.noarch. 17 MB/s | 384 kB 00:00 2023-06-23T00:54:04.198 INFO:teuthology.orchestra.run.smithi131.stdout:(47/120): python3-google-auth-1.1.1-10.el8.noar 4.3 MB/s | 81 kB 00:00 2023-06-23T00:54:04.218 INFO:teuthology.orchestra.run.smithi131.stdout:(48/120): python3-jaraco-6.2-6.el8.noarch.rpm 636 kB/s | 11 kB 00:00 2023-06-23T00:54:04.234 INFO:teuthology.orchestra.run.smithi131.stdout:(49/120): python3-jaraco-functools-2.0-4.el8.no 988 kB/s | 18 kB 00:00 2023-06-23T00:54:04.276 INFO:teuthology.orchestra.run.smithi131.stdout:(50/120): python3-kubernetes-11.0.0-6.el8.noarc 28 MB/s | 1.2 MB 00:00 2023-06-23T00:54:04.296 INFO:teuthology.orchestra.run.smithi131.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 2.6 MB/s | 49 kB 00:00 2023-06-23T00:54:04.313 INFO:teuthology.orchestra.run.smithi131.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 3.1 MB/s | 59 kB 00:00 2023-06-23T00:54:04.331 INFO:teuthology.orchestra.run.smithi131.stdout:(53/120): python3-natsort-7.1.1-2.el8.noarch.rp 3.2 MB/s | 60 kB 00:00 2023-06-23T00:54:04.353 INFO:teuthology.orchestra.run.smithi131.stdout:(54/120): python3-pecan-1.3.2-9.el8.noarch.rpm 13 MB/s | 283 kB 00:00 2023-06-23T00:54:04.371 INFO:teuthology.orchestra.run.smithi131.stdout:(55/120): python3-portend-2.6-1.el8.noarch.rpm 892 kB/s | 16 kB 00:00 2023-06-23T00:54:04.388 INFO:teuthology.orchestra.run.smithi131.stdout:(56/120): python3-repoze-lru-0.7-6.el8.noarch.r 1.8 MB/s | 34 kB 00:00 2023-06-23T00:54:04.408 INFO:teuthology.orchestra.run.smithi131.stdout:(57/120): python3-routes-2.4.1-12.el8.noarch.rp 9.7 MB/s | 196 kB 00:00 2023-06-23T00:54:04.427 INFO:teuthology.orchestra.run.smithi131.stdout:(58/120): python3-rsa-4.8-1.el8.noarch.rpm 3.3 MB/s | 61 kB 00:00 2023-06-23T00:54:04.445 INFO:teuthology.orchestra.run.smithi131.stdout:(59/120): python3-simplegeneric-0.8.1-17.el8.no 1.0 MB/s | 19 kB 00:00 2023-06-23T00:54:04.483 INFO:teuthology.orchestra.run.smithi131.stdout:(60/120): ceph-mgr-diskprediction-local-17.2.6- 8.4 MB/s | 7.4 MB 00:00 2023-06-23T00:54:04.483 INFO:teuthology.orchestra.run.smithi131.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 648 kB/s | 24 kB 00:00 2023-06-23T00:54:04.501 INFO:teuthology.orchestra.run.smithi131.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 1.6 MB/s | 29 kB 00:00 2023-06-23T00:54:04.521 INFO:teuthology.orchestra.run.smithi131.stdout:(63/120): python3-waitress-1.4.3-1.el8.noarch.r 12 MB/s | 255 kB 00:00 2023-06-23T00:54:04.538 INFO:teuthology.orchestra.run.smithi131.stdout:(64/120): python3-trustme-0.6.0-4.el8.noarch.rp 498 kB/s | 27 kB 00:00 2023-06-23T00:54:04.541 INFO:teuthology.orchestra.run.smithi131.stdout:(65/120): python3-webob-1.8.5-1.el8.1.noarch.rp 12 MB/s | 251 kB 00:00 2023-06-23T00:54:04.560 INFO:teuthology.orchestra.run.smithi131.stdout:(66/120): python3-webtest-2.0.33-1.el8.noarch.r 4.5 MB/s | 85 kB 00:00 2023-06-23T00:54:04.576 INFO:teuthology.orchestra.run.smithi131.stdout:(67/120): python3-websocket-client-0.56.0-5.el8 1.6 MB/s | 61 kB 00:00 2023-06-23T00:54:04.578 INFO:teuthology.orchestra.run.smithi131.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 1.3 MB/s | 23 kB 00:00 2023-06-23T00:54:04.582 INFO:teuthology.orchestra.run.smithi131.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 17 MB/s | 70 kB 00:00 2023-06-23T00:54:04.668 INFO:teuthology.orchestra.run.smithi131.stdout:(70/120): thrift-0.13.0-2.el8.x86_64.rpm 19 MB/s | 1.7 MB 00:00 2023-06-23T00:54:04.698 INFO:teuthology.orchestra.run.smithi131.stdout:(71/120): python3-prettytable-0.7.2-14.el8.noar 1.4 MB/s | 44 kB 00:00 2023-06-23T00:54:04.713 INFO:teuthology.orchestra.run.smithi131.stdout:(72/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.7 MB/s | 27 kB 00:00 2023-06-23T00:54:04.731 INFO:teuthology.orchestra.run.smithi131.stdout:(73/120): jq-1.5-12.el8.x86_64.rpm 9.6 MB/s | 161 kB 00:00 2023-06-23T00:54:04.748 INFO:teuthology.orchestra.run.smithi131.stdout:(74/120): flex-2.6.1-9.el8.x86_64.rpm 18 MB/s | 320 kB 00:00 2023-06-23T00:54:05.265 INFO:teuthology.orchestra.run.smithi131.stdout:(75/120): ceph-test-17.2.6-719.g42ddc46e.el8.x8 7.1 MB/s | 52 MB 00:07 2023-06-23T00:54:05.323 INFO:teuthology.orchestra.run.smithi131.stdout:(76/120): python3-werkzeug-0.12.2-4.el8.noarch. 796 kB/s | 457 kB 00:00 2023-06-23T00:54:05.458 INFO:teuthology.orchestra.run.smithi131.stdout:(77/120): python3-scipy-1.0.0-20.module+el8.1.0 17 MB/s | 15 MB 00:00 2023-06-23T00:54:05.544 INFO:teuthology.orchestra.run.smithi131.stdout:(78/120): python3-markupsafe-0.23-19.el8.x86_64 182 kB/s | 39 kB 00:00 2023-06-23T00:54:05.622 INFO:teuthology.orchestra.run.smithi131.stdout:(79/120): bison-3.0.4-10.el8.x86_64.rpm 4.1 MB/s | 688 kB 00:00 2023-06-23T00:54:05.688 INFO:teuthology.orchestra.run.smithi131.stdout:(80/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.7 MB/s | 259 kB 00:00 2023-06-23T00:54:05.692 INFO:teuthology.orchestra.run.smithi131.stdout:(81/120): python3-numpy-f2py-1.14.3-9.el8.x86_6 3.1 MB/s | 225 kB 00:00 2023-06-23T00:54:05.709 INFO:teuthology.orchestra.run.smithi131.stdout:(82/120): python3-pyasn1-0.3.7-6.el8.noarch.rpm 6.3 MB/s | 126 kB 00:00 2023-06-23T00:54:05.729 INFO:teuthology.orchestra.run.smithi131.stdout:(83/120): oniguruma-6.8.2-2.el8.x86_64.rpm 8.7 MB/s | 187 kB 00:00 2023-06-23T00:54:05.753 INFO:teuthology.orchestra.run.smithi131.stdout:(84/120): python3-babel-2.5.1-5.el8.noarch.rpm 9.8 MB/s | 4.8 MB 00:00 2023-06-23T00:54:05.761 INFO:teuthology.orchestra.run.smithi131.stdout:(85/120): python3-jinja2-2.10.1-2.el8_0.noarch. 17 MB/s | 538 kB 00:00 2023-06-23T00:54:05.780 INFO:teuthology.orchestra.run.smithi131.stdout:(86/120): python3-numpy-1.14.3-9.el8.x86_64.rpm 43 MB/s | 3.7 MB 00:00 2023-06-23T00:54:05.783 INFO:teuthology.orchestra.run.smithi131.stdout:(87/120): python3-mako-1.0.6-13.el8.noarch.rpm 7.1 MB/s | 157 kB 00:00 2023-06-23T00:54:05.785 INFO:teuthology.orchestra.run.smithi131.stdout:(88/120): socat-1.7.3.3-2.el8.x86_64.rpm 9.3 MB/s | 302 kB 00:00 2023-06-23T00:54:05.795 INFO:teuthology.orchestra.run.smithi131.stdout:(89/120): python3-pyasn1-modules-0.3.7-6.el8.no 7.1 MB/s | 110 kB 00:00 2023-06-23T00:54:05.803 INFO:teuthology.orchestra.run.smithi131.stdout:(90/120): python3-jsonpointer-1.10-11.el8.noarc 1.2 MB/s | 20 kB 00:00 2023-06-23T00:54:05.804 INFO:teuthology.orchestra.run.smithi131.stdout:(91/120): librdkafka-0.11.4-1.el8.x86_64.rpm 16 MB/s | 353 kB 00:00 2023-06-23T00:54:05.818 INFO:teuthology.orchestra.run.smithi131.stdout:(92/120): libtool-2.4.6-25.el8.x86_64.rpm 32 MB/s | 709 kB 00:00 2023-06-23T00:54:05.819 INFO:teuthology.orchestra.run.smithi131.stdout:(93/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.1 MB/s | 54 kB 00:00 2023-06-23T00:54:05.844 INFO:teuthology.orchestra.run.smithi131.stdout:(94/120): automake-1.16.1-7.el8.noarch.rpm 28 MB/s | 713 kB 00:00 2023-06-23T00:54:05.857 INFO:teuthology.orchestra.run.smithi131.stdout:(95/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 1.4 MB/s | 19 kB 00:00 2023-06-23T00:54:05.869 INFO:teuthology.orchestra.run.smithi131.stdout:(96/120): python3-jwt-1.6.1-2.el8.noarch.rpm 3.5 MB/s | 43 kB 00:00 2023-06-23T00:54:05.885 INFO:teuthology.orchestra.run.smithi131.stdout:(97/120): openblas-threads-0.3.12-1.el8.x86_64. 59 MB/s | 4.7 MB 00:00 2023-06-23T00:54:05.887 INFO:teuthology.orchestra.run.smithi131.stdout:(98/120): python3-oauthlib-2.1.0-1.el8.noarch.r 8.4 MB/s | 155 kB 00:00 2023-06-23T00:54:05.908 INFO:teuthology.orchestra.run.smithi131.stdout:(99/120): openblas-0.3.12-1.el8.x86_64.rpm 51 MB/s | 4.6 MB 00:00 2023-06-23T00:54:05.910 INFO:teuthology.orchestra.run.smithi131.stdout:(100/120): keyutils-libs-devel-1.5.10-6.el8.x86 2.2 MB/s | 48 kB 00:00 2023-06-23T00:54:05.911 INFO:teuthology.orchestra.run.smithi131.stdout:(101/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 8.3 MB/s | 220 kB 00:00 2023-06-23T00:54:05.925 INFO:teuthology.orchestra.run.smithi131.stdout:(102/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 14 MB/s | 229 kB 00:00 2023-06-23T00:54:05.925 INFO:teuthology.orchestra.run.smithi131.stdout:(103/120): python3-requests-oauthlib-1.0.0-1.el 2.6 MB/s | 43 kB 00:00 2023-06-23T00:54:05.941 INFO:teuthology.orchestra.run.smithi131.stdout:(104/120): libverto-devel-0.3.0-5.el8.x86_64.rp 1.1 MB/s | 18 kB 00:00 2023-06-23T00:54:05.951 INFO:teuthology.orchestra.run.smithi131.stdout:(105/120): openssl-devel-1.1.1g-15.el8_3.x86_64 59 MB/s | 2.3 MB 00:00 2023-06-23T00:54:05.953 INFO:teuthology.orchestra.run.smithi131.stdout:(106/120): pcre2-devel-10.32-2.el8.x86_64.rpm 22 MB/s | 605 kB 00:00 2023-06-23T00:54:05.954 INFO:teuthology.orchestra.run.smithi131.stdout:(107/120): fuse-2.9.7-12.el8.x86_64.rpm 6.1 MB/s | 83 kB 00:00 2023-06-23T00:54:05.966 INFO:teuthology.orchestra.run.smithi131.stdout:(108/120): elfutils-libelf-devel-0.182-3.el8.x8 4.0 MB/s | 59 kB 00:00 2023-06-23T00:54:05.970 INFO:teuthology.orchestra.run.smithi131.stdout:(109/120): libquadmath-8.4.1-1.el8.x86_64.rpm 9.9 MB/s | 169 kB 00:00 2023-06-23T00:54:05.970 INFO:teuthology.orchestra.run.smithi131.stdout:(110/120): libcom_err-devel-1.45.6-1.el8.x86_64 2.3 MB/s | 38 kB 00:00 2023-06-23T00:54:05.982 INFO:teuthology.orchestra.run.smithi131.stdout:(111/120): libsepol-devel-2.9-2.el8.x86_64.rpm 5.0 MB/s | 86 kB 00:00 2023-06-23T00:54:05.984 INFO:teuthology.orchestra.run.smithi131.stdout:(112/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.4 MB/s | 47 kB 00:00 2023-06-23T00:54:05.992 INFO:teuthology.orchestra.run.smithi131.stdout:(113/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.6 MB/s | 58 kB 00:00 2023-06-23T00:54:06.004 INFO:teuthology.orchestra.run.smithi131.stdout:(114/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 26 MB/s | 559 kB 00:00 2023-06-23T00:54:06.007 INFO:teuthology.orchestra.run.smithi131.stdout:(115/120): libgfortran-8.4.1-1.el8.x86_64.rpm 28 MB/s | 639 kB 00:00 2023-06-23T00:54:06.008 INFO:teuthology.orchestra.run.smithi131.stdout:(116/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 4.2 MB/s | 70 kB 00:00 2023-06-23T00:54:06.019 INFO:teuthology.orchestra.run.smithi131.stdout:(117/120): libselinux-devel-2.9-5.el8.x86_64.rp 13 MB/s | 200 kB 00:00 2023-06-23T00:54:06.025 INFO:teuthology.orchestra.run.smithi131.stdout:(118/120): libuuid-devel-2.32.1-27.el8.x86_64.r 5.5 MB/s | 97 kB 00:00 2023-06-23T00:54:06.331 INFO:teuthology.orchestra.run.smithi131.stdout:(119/120): librados2-17.2.6-719.g42ddc46e.el8.x 11 MB/s | 3.5 MB 00:00 2023-06-23T00:54:06.483 INFO:teuthology.orchestra.run.smithi131.stdout:(120/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 7.7 MB/s | 3.6 MB 00:00 2023-06-23T00:54:06.487 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2023-06-23T00:54:06.488 INFO:teuthology.orchestra.run.smithi131.stdout:Total 11 MB/s | 201 MB 00:17 2023-06-23T00:54:06.534 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2023-06-23T00:54:06.646 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2023-06-23T00:54:06.647 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2023-06-23T00:54:07.944 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-06-23T00:54:08.113 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2023-06-23T00:54:08.118 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2023-06-23T00:54:08.126 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-23T00:54:08.360 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/122 2023-06-23T00:54:08.535 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T00:54:08.648 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T00:54:08.860 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T00:54:09.279 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T00:54:09.504 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T00:54:09.933 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T00:54:10.485 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:54:10.601 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:54:10.787 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:54:10.895 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:54:11.249 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T00:54:11.901 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-23T00:54:12.067 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:54:12.184 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:54:12.405 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-23T00:54:12.592 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-23T00:54:12.822 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:54:12.938 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:54:12.986 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2023-06-23T00:54:13.101 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T00:54:13.153 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T00:54:13.175 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-23T00:54:13.304 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 15/122 2023-06-23T00:54:13.384 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/122 2023-06-23T00:54:13.476 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T00:54:13.578 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T00:54:13.614 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T00:54:13.678 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T00:54:13.767 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T00:54:13.903 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T00:54:14.272 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T00:54:14.313 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T00:54:14.461 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T00:54:14.467 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T00:54:14.852 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T00:54:15.003 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T00:54:15.045 INFO:teuthology.orchestra.run.smithi131.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:54:15.171 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:54:15.175 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T00:54:15.344 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:54:15.440 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:54:15.787 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T00:54:15.887 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T00:54:16.170 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 20/122 2023-06-23T00:54:16.349 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 21/122 2023-06-23T00:54:16.437 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-23T00:54:16.529 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 22/122 2023-06-23T00:54:16.610 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:54:16.722 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:54:16.806 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/122 2023-06-23T00:54:17.060 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/122 2023-06-23T00:54:17.287 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-23T00:54:17.334 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-23T00:54:17.465 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-23T00:54:17.514 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-23T00:54:17.605 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-23T00:54:17.711 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:54:17.827 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:54:17.848 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T00:54:17.994 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T00:54:18.060 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T00:54:18.192 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 15/122 2023-06-23T00:54:18.311 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T00:54:18.445 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T00:54:18.602 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T00:54:18.868 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/122 2023-06-23T00:54:19.151 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T00:54:19.195 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/122 2023-06-23T00:54:19.319 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T00:54:19.506 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/122 2023-06-23T00:54:19.713 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 32/122 2023-06-23T00:54:19.849 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T00:54:19.999 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 33/122 2023-06-23T00:54:20.035 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T00:54:20.224 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T00:54:20.748 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T00:54:20.988 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 20/122 2023-06-23T00:54:21.183 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 21/122 2023-06-23T00:54:21.347 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 22/122 2023-06-23T00:54:21.605 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/122 2023-06-23T00:54:21.911 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/122 2023-06-23T00:54:22.146 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-23T00:54:22.283 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T00:54:22.327 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-23T00:54:22.447 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-23T00:54:22.636 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 35/122 2023-06-23T00:54:22.674 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T00:54:22.838 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-23T00:54:23.027 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-23T00:54:23.275 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-23T00:54:23.462 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-23T00:54:23.656 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-23T00:54:23.676 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/122 2023-06-23T00:54:23.838 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 41/122 2023-06-23T00:54:23.994 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 42/122 2023-06-23T00:54:23.997 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/122 2023-06-23T00:54:24.166 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 43/122 2023-06-23T00:54:24.291 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/122 2023-06-23T00:54:24.325 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fuse-2.9.7-12.el8.x86_64 44/122 2023-06-23T00:54:24.502 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 45/122 2023-06-23T00:54:24.566 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 32/122 2023-06-23T00:54:24.703 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 46/122 2023-06-23T00:54:24.841 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 33/122 2023-06-23T00:54:24.859 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/122 2023-06-23T00:54:25.059 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T00:54:25.208 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/122 2023-06-23T00:54:25.494 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 49/122 2023-06-23T00:54:25.796 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 50/122 2023-06-23T00:54:26.097 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 51/122 2023-06-23T00:54:26.696 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : automake-1.16.1-7.el8.noarch 52/122 2023-06-23T00:54:26.881 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/122 2023-06-23T00:54:27.119 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/122 2023-06-23T00:54:27.126 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T00:54:27.301 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 55/122 2023-06-23T00:54:27.446 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 35/122 2023-06-23T00:54:27.456 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T00:54:27.536 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T00:54:27.664 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-23T00:54:27.748 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : jq-1.5-12.el8.x86_64 57/122 2023-06-23T00:54:27.880 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-23T00:54:27.931 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 58/122 2023-06-23T00:54:28.086 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-23T00:54:28.089 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-23T00:54:28.252 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-23T00:54:28.255 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-23T00:54:28.441 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-23T00:54:28.512 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-23T00:54:28.624 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 41/122 2023-06-23T00:54:28.714 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-23T00:54:28.789 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 42/122 2023-06-23T00:54:28.871 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-23T00:54:28.960 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 43/122 2023-06-23T00:54:29.060 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-23T00:54:29.111 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : fuse-2.9.7-12.el8.x86_64 44/122 2023-06-23T00:54:29.216 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-23T00:54:29.255 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 45/122 2023-06-23T00:54:29.415 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-23T00:54:29.474 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 46/122 2023-06-23T00:54:29.620 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/122 2023-06-23T00:54:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-23T00:54:29.864 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-23T00:54:29.917 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/122 2023-06-23T00:54:30.050 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-23T00:54:30.171 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 49/122 2023-06-23T00:54:30.236 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-23T00:54:30.430 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 50/122 2023-06-23T00:54:30.459 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-23T00:54:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-23T00:54:30.758 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 51/122 2023-06-23T00:54:30.877 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-23T00:54:30.974 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : automake-1.16.1-7.el8.noarch 52/122 2023-06-23T00:54:31.141 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-23T00:54:31.166 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/122 2023-06-23T00:54:31.354 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/122 2023-06-23T00:54:31.379 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-23T00:54:31.537 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 55/122 2023-06-23T00:54:31.587 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-23T00:54:31.708 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T00:54:31.787 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T00:54:31.827 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-23T00:54:31.958 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : jq-1.5-12.el8.x86_64 57/122 2023-06-23T00:54:32.127 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 58/122 2023-06-23T00:54:32.283 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-23T00:54:32.446 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-23T00:54:32.689 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-23T00:54:32.917 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-23T00:54:32.985 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-23T00:54:33.073 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-23T00:54:33.221 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-23T00:54:33.278 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-23T00:54:33.352 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-23T00:54:33.528 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-23T00:54:33.543 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-23T00:54:33.744 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-23T00:54:33.845 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-23T00:54:33.959 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-23T00:54:34.136 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-23T00:54:34.139 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-23T00:54:34.272 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-23T00:54:34.486 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-23T00:54:34.502 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-23T00:54:34.711 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-23T00:54:34.746 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-23T00:54:34.905 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-23T00:54:34.938 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-23T00:54:35.130 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-23T00:54:35.177 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-23T00:54:35.423 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-23T00:54:35.598 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-23T00:54:35.838 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-23T00:54:36.987 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-23T00:54:37.297 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-23T00:54:37.534 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-23T00:54:37.875 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-23T00:54:38.134 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-23T00:54:38.454 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-23T00:54:38.706 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-23T00:54:38.966 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-23T00:54:39.175 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-23T00:54:40.038 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:54:40.254 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:54:40.846 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:54:41.081 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T00:54:41.205 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T00:54:41.344 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T00:54:44.034 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:54:44.273 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:54:44.869 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:54:45.118 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T00:54:45.226 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T00:54:45.356 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T00:54:59.200 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T00:54:59.201 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /sys 2023-06-23T00:54:59.201 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /proc 2023-06-23T00:54:59.201 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /dev 2023-06-23T00:54:59.201 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /run 2023-06-23T00:54:59.201 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /mnt 2023-06-23T00:54:59.202 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /var/tmp 2023-06-23T00:54:59.202 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /home 2023-06-23T00:54:59.202 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /tmp 2023-06-23T00:54:59.202 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /dev 2023-06-23T00:54:59.202 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:54:59.557 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T00:55:01.044 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T00:55:01.209 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T00:55:01.437 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T00:55:01.574 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T00:55:01.965 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T00:55:02.115 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T00:55:02.133 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T00:55:03.477 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T00:55:03.478 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /sys 2023-06-23T00:55:03.478 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /proc 2023-06-23T00:55:03.478 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /dev 2023-06-23T00:55:03.479 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /run 2023-06-23T00:55:03.479 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /mnt 2023-06-23T00:55:03.479 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /var/tmp 2023-06-23T00:55:03.479 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /home 2023-06-23T00:55:03.480 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /tmp 2023-06-23T00:55:03.480 INFO:teuthology.orchestra.run.smithi131.stdout:skipping the directory /dev 2023-06-23T00:55:03.480 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:55:03.837 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T00:55:04.886 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T00:55:05.021 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T00:55:05.229 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-23T00:55:05.324 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T00:55:05.411 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-23T00:55:05.490 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T00:55:05.580 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T00:55:05.712 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T00:55:05.796 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T00:55:05.862 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T00:55:06.000 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T00:55:06.251 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T00:55:06.397 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T00:55:06.412 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T00:55:06.971 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T00:55:07.172 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T00:55:07.270 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T00:55:07.630 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-23T00:55:07.833 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T00:55:07.870 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T00:55:08.047 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T00:55:08.058 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T00:55:09.071 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T00:55:09.252 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T00:55:09.425 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-23T00:55:09.598 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-23T00:55:09.660 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-23T00:55:09.777 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T00:55:09.874 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T00:55:09.969 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T00:55:09.992 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T00:55:10.130 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T00:55:10.173 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T00:55:10.197 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T00:55:10.380 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T00:55:10.729 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T00:55:10.905 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T00:55:11.167 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T00:55:11.344 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T00:55:11.441 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T00:55:11.770 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-23T00:55:11.914 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T00:55:11.950 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T00:55:12.135 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T00:55:12.147 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T00:55:13.774 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-23T00:55:13.989 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T00:55:14.078 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T00:55:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T00:55:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T00:55:14.916 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T00:55:14.917 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T00:55:14.952 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T00:55:15.847 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T00:55:16.164 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-23T00:55:16.320 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T00:55:17.035 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T00:55:17.378 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 110/122 2023-06-23T00:55:17.557 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 111/122 2023-06-23T00:55:17.707 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 112/122 2023-06-23T00:55:17.866 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-23T00:55:18.049 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-23T00:55:18.195 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-23T00:55:18.369 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-23T00:55:18.512 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-23T00:55:18.730 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 118/122 2023-06-23T00:55:18.911 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T00:55:18.963 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T00:55:19.128 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T00:55:19.137 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T00:55:19.137 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T00:55:19.398 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T00:55:19.399 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:55:19.884 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T00:55:19.902 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:55:19.902 INFO:teuthology.orchestra.run.smithi092.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-23T00:55:19.902 INFO:teuthology.orchestra.run.smithi092.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-23T00:55:19.902 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:55:20.205 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-23T00:55:20.255 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-23T00:55:20.255 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-23T00:55:20.255 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-23T00:55:20.255 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T00:55:20.255 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-23T00:55:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:55:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:55:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T00:55:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-23T00:55:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:55:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-23T00:55:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-23T00:55:20.257 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:55:20.257 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-23T00:55:20.257 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-23T00:55:20.257 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-23T00:55:20.257 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-23T00:55:20.257 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-23T00:55:20.257 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-23T00:55:20.257 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-23T00:55:20.258 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-23T00:55:20.258 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-23T00:55:20.258 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-23T00:55:20.258 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-23T00:55:20.258 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-23T00:55:20.258 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-23T00:55:20.258 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-23T00:55:20.258 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T00:55:20.259 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-23T00:55:20.259 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-23T00:55:20.259 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-23T00:55:20.259 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-23T00:55:20.259 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-23T00:55:20.259 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-23T00:55:20.259 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-23T00:55:20.260 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-23T00:55:20.260 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-23T00:55:20.260 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-23T00:55:20.260 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-23T00:55:20.260 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-23T00:55:20.260 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-23T00:55:20.260 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-23T00:55:20.260 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-23T00:55:20.261 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-23T00:55:20.261 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-23T00:55:20.261 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-23T00:55:20.261 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-23T00:55:20.261 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-23T00:55:20.261 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-23T00:55:20.261 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-23T00:55:20.262 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-23T00:55:20.262 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-23T00:55:20.262 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-23T00:55:20.262 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-23T00:55:20.262 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-23T00:55:20.262 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-23T00:55:20.262 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-23T00:55:20.262 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-23T00:55:20.263 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-23T00:55:20.263 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-23T00:55:20.263 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-23T00:55:20.263 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-23T00:55:20.263 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-23T00:55:20.264 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-23T00:55:20.264 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-23T00:55:20.264 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-23T00:55:20.264 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-23T00:55:20.264 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-23T00:55:20.264 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-23T00:55:20.264 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-23T00:55:20.264 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-23T00:55:20.265 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 72/122 2023-06-23T00:55:20.265 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/122 2023-06-23T00:55:20.265 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/122 2023-06-23T00:55:20.265 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : jq-1.5-12.el8.x86_64 75/122 2023-06-23T00:55:20.265 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flex-2.6.1-9.el8.x86_64 76/122 2023-06-23T00:55:20.265 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/122 2023-06-23T00:55:20.265 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 78/122 2023-06-23T00:55:20.265 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 79/122 2023-06-23T00:55:20.266 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/122 2023-06-23T00:55:20.266 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 81/122 2023-06-23T00:55:20.266 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 82/122 2023-06-23T00:55:20.266 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 83/122 2023-06-23T00:55:20.266 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 84/122 2023-06-23T00:55:20.266 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/122 2023-06-23T00:55:20.266 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 86/122 2023-06-23T00:55:20.267 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 87/122 2023-06-23T00:55:20.267 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 88/122 2023-06-23T00:55:20.267 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 89/122 2023-06-23T00:55:20.267 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 90/122 2023-06-23T00:55:20.267 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 91/122 2023-06-23T00:55:20.267 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 92/122 2023-06-23T00:55:20.267 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 93/122 2023-06-23T00:55:20.267 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 94/122 2023-06-23T00:55:20.268 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/122 2023-06-23T00:55:20.268 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : automake-1.16.1-7.el8.noarch 96/122 2023-06-23T00:55:20.268 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 97/122 2023-06-23T00:55:20.268 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/122 2023-06-23T00:55:20.268 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/122 2023-06-23T00:55:20.268 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 100/122 2023-06-23T00:55:20.269 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 101/122 2023-06-23T00:55:20.269 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 102/122 2023-06-23T00:55:20.269 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/122 2023-06-23T00:55:20.269 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 104/122 2023-06-23T00:55:20.269 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/122 2023-06-23T00:55:20.269 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 106/122 2023-06-23T00:55:20.269 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 107/122 2023-06-23T00:55:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 108/122 2023-06-23T00:55:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 109/122 2023-06-23T00:55:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 110/122 2023-06-23T00:55:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 111/122 2023-06-23T00:55:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 112/122 2023-06-23T00:55:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 113/122 2023-06-23T00:55:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 114/122 2023-06-23T00:55:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 115/122 2023-06-23T00:55:20.271 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 116/122 2023-06-23T00:55:20.271 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 117/122 2023-06-23T00:55:20.271 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 118/122 2023-06-23T00:55:20.271 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-23T00:55:20.271 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-23T00:55:20.271 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-23T00:55:20.359 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T00:55:21.058 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T00:55:21.398 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 110/122 2023-06-23T00:55:21.562 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 111/122 2023-06-23T00:55:21.712 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 112/122 2023-06-23T00:55:21.847 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-23T00:55:21.988 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-23T00:55:22.143 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-23T00:55:22.313 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-23T00:55:22.460 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-23T00:55:22.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:55:22.547 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:43 ago on Fri 23 Jun 2023 12:53:39 AM UTC. 2023-06-23T00:55:22.646 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 118/122 2023-06-23T00:55:22.817 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T00:55:22.872 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T00:55:23.017 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T00:55:23.026 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T00:55:23.026 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T00:55:23.228 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T00:55:23.229 INFO:teuthology.orchestra.run.smithi131.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:55:23.724 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:55:23.725 INFO:teuthology.orchestra.run.smithi131.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-23T00:55:23.725 INFO:teuthology.orchestra.run.smithi131.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-23T00:55:23.725 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:55:24.093 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-23T00:55:24.093 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-23T00:55:24.094 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-23T00:55:24.094 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T00:55:24.094 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-23T00:55:24.094 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:55:24.094 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:55:24.094 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T00:55:24.094 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-23T00:55:24.095 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:55:24.095 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-23T00:55:24.095 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-23T00:55:24.095 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:55:24.095 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-23T00:55:24.095 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-23T00:55:24.095 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-23T00:55:24.095 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-23T00:55:24.096 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-23T00:55:24.096 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-23T00:55:24.096 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-23T00:55:24.096 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-23T00:55:24.096 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-23T00:55:24.096 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-23T00:55:24.096 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-23T00:55:24.097 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-23T00:55:24.103 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-23T00:55:24.103 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-23T00:55:24.103 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T00:55:24.103 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-23T00:55:24.104 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-23T00:55:24.104 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-23T00:55:24.104 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-23T00:55:24.104 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-23T00:55:24.104 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-23T00:55:24.104 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-23T00:55:24.105 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-23T00:55:24.105 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-23T00:55:24.105 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-23T00:55:24.105 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-23T00:55:24.105 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-23T00:55:24.105 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-23T00:55:24.105 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-23T00:55:24.106 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-23T00:55:24.106 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-23T00:55:24.106 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-23T00:55:24.106 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-23T00:55:24.106 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-23T00:55:24.106 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-23T00:55:24.106 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-23T00:55:24.106 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-23T00:55:24.107 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-23T00:55:24.107 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-23T00:55:24.107 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-23T00:55:24.107 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-23T00:55:24.107 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-23T00:55:24.107 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-23T00:55:24.107 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-23T00:55:24.107 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-23T00:55:24.108 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-23T00:55:24.108 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-23T00:55:24.108 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-23T00:55:24.108 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-23T00:55:24.108 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-23T00:55:24.108 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-23T00:55:24.108 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-23T00:55:24.109 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-23T00:55:24.109 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-23T00:55:24.109 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-23T00:55:24.109 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-23T00:55:24.109 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-23T00:55:24.109 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-23T00:55:24.109 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 72/122 2023-06-23T00:55:24.110 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/122 2023-06-23T00:55:24.110 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/122 2023-06-23T00:55:24.110 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : jq-1.5-12.el8.x86_64 75/122 2023-06-23T00:55:24.110 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : flex-2.6.1-9.el8.x86_64 76/122 2023-06-23T00:55:24.110 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/122 2023-06-23T00:55:24.110 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 78/122 2023-06-23T00:55:24.110 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 79/122 2023-06-23T00:55:24.110 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/122 2023-06-23T00:55:24.111 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 81/122 2023-06-23T00:55:24.111 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 82/122 2023-06-23T00:55:24.111 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 83/122 2023-06-23T00:55:24.111 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 84/122 2023-06-23T00:55:24.111 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/122 2023-06-23T00:55:24.111 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 86/122 2023-06-23T00:55:24.111 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 87/122 2023-06-23T00:55:24.111 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 88/122 2023-06-23T00:55:24.112 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 89/122 2023-06-23T00:55:24.112 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 90/122 2023-06-23T00:55:24.112 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 91/122 2023-06-23T00:55:24.112 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 92/122 2023-06-23T00:55:24.112 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 93/122 2023-06-23T00:55:24.112 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 94/122 2023-06-23T00:55:24.112 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/122 2023-06-23T00:55:24.113 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : automake-1.16.1-7.el8.noarch 96/122 2023-06-23T00:55:24.113 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 97/122 2023-06-23T00:55:24.113 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/122 2023-06-23T00:55:24.113 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/122 2023-06-23T00:55:24.113 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 100/122 2023-06-23T00:55:24.113 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 101/122 2023-06-23T00:55:24.113 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 102/122 2023-06-23T00:55:24.113 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/122 2023-06-23T00:55:24.114 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 104/122 2023-06-23T00:55:24.114 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/122 2023-06-23T00:55:24.114 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 106/122 2023-06-23T00:55:24.114 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 107/122 2023-06-23T00:55:24.114 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 108/122 2023-06-23T00:55:24.114 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 109/122 2023-06-23T00:55:24.114 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 110/122 2023-06-23T00:55:24.115 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 111/122 2023-06-23T00:55:24.115 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 112/122 2023-06-23T00:55:24.115 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 113/122 2023-06-23T00:55:24.115 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 114/122 2023-06-23T00:55:24.115 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 115/122 2023-06-23T00:55:24.115 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 116/122 2023-06-23T00:55:24.115 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 117/122 2023-06-23T00:55:24.115 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 118/122 2023-06-23T00:55:24.116 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-23T00:55:24.116 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-23T00:55:24.116 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-23T00:55:24.949 INFO:teuthology.orchestra.run.smithi092.stdout:Installed products updated. 2023-06-23T00:55:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:55:25.072 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2023-06-23T00:55:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.072 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-06-23T00:55:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T00:55:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T00:55:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-23T00:55:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T00:55:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-23T00:55:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T00:55:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: jq-1.5-12.el8.x86_64 2023-06-23T00:55:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-23T00:55:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-23T00:55:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T00:55:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-23T00:55:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-23T00:55:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T00:55:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-23T00:55:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T00:55:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-23T00:55:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-23T00:55:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-23T00:55:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T00:55:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T00:55:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-23T00:55:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-23T00:55:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T00:55:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T00:55:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-23T00:55:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-23T00:55:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-23T00:55:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-23T00:55:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-23T00:55:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-23T00:55:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T00:55:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-23T00:55:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T00:55:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T00:55:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T00:55:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T00:55:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T00:55:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T00:55:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T00:55:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T00:55:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T00:55:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-23T00:55:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T00:55:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T00:55:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T00:55:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T00:55:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T00:55:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-23T00:55:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T00:55:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T00:55:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T00:55:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-23T00:55:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-23T00:55:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T00:55:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T00:55:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T00:55:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T00:55:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T00:55:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T00:55:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T00:55:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T00:55:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T00:55:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T00:55:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T00:55:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-23T00:55:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T00:55:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T00:55:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T00:55:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T00:55:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T00:55:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T00:55:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T00:55:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T00:55:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T00:55:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T00:55:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-23T00:55:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T00:55:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-23T00:55:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T00:55:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-23T00:55:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T00:55:25.089 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-06-23T00:55:25.305 DEBUG:teuthology.parallel:result is None 2023-06-23T00:55:26.301 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:55:26.301 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:01:47 ago on Fri 23 Jun 2023 12:53:39 AM UTC. 2023-06-23T00:55:28.679 INFO:teuthology.orchestra.run.smithi131.stdout:Installed products updated. 2023-06-23T00:55:28.801 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:55:28.801 INFO:teuthology.orchestra.run.smithi131.stdout:Upgraded: 2023-06-23T00:55:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.802 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2023-06-23T00:55:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T00:55:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T00:55:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.802 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.803 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.804 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:28.805 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.805 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.805 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.805 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:28.805 INFO:teuthology.orchestra.run.smithi131.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:28.805 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.805 INFO:teuthology.orchestra.run.smithi131.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:55:28.806 INFO:teuthology.orchestra.run.smithi131.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-23T00:55:28.806 INFO:teuthology.orchestra.run.smithi131.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T00:55:28.806 INFO:teuthology.orchestra.run.smithi131.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-23T00:55:28.807 INFO:teuthology.orchestra.run.smithi131.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T00:55:28.807 INFO:teuthology.orchestra.run.smithi131.stdout: jq-1.5-12.el8.x86_64 2023-06-23T00:55:28.807 INFO:teuthology.orchestra.run.smithi131.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-23T00:55:28.807 INFO:teuthology.orchestra.run.smithi131.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-23T00:55:28.808 INFO:teuthology.orchestra.run.smithi131.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T00:55:28.808 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.808 INFO:teuthology.orchestra.run.smithi131.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.808 INFO:teuthology.orchestra.run.smithi131.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.808 INFO:teuthology.orchestra.run.smithi131.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-23T00:55:28.808 INFO:teuthology.orchestra.run.smithi131.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-23T00:55:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T00:55:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-23T00:55:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T00:55:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-23T00:55:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.809 INFO:teuthology.orchestra.run.smithi131.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-23T00:55:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-23T00:55:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T00:55:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T00:55:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-23T00:55:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-23T00:55:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T00:55:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T00:55:28.810 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-23T00:55:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-23T00:55:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-23T00:55:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-23T00:55:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-23T00:55:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-23T00:55:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T00:55:28.811 INFO:teuthology.orchestra.run.smithi131.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-23T00:55:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T00:55:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T00:55:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T00:55:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.812 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T00:55:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T00:55:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T00:55:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T00:55:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T00:55:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T00:55:28.813 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-23T00:55:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T00:55:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T00:55:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T00:55:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T00:55:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T00:55:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-23T00:55:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T00:55:28.814 INFO:teuthology.orchestra.run.smithi131.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T00:55:28.815 INFO:teuthology.orchestra.run.smithi131.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T00:55:28.815 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-23T00:55:28.815 INFO:teuthology.orchestra.run.smithi131.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-23T00:55:28.815 INFO:teuthology.orchestra.run.smithi131.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T00:55:28.815 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T00:55:28.815 INFO:teuthology.orchestra.run.smithi131.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T00:55:28.815 INFO:teuthology.orchestra.run.smithi131.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T00:55:28.816 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T00:55:28.816 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T00:55:28.816 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T00:55:28.816 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.816 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.816 INFO:teuthology.orchestra.run.smithi131.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T00:55:28.816 INFO:teuthology.orchestra.run.smithi131.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T00:55:28.816 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.817 INFO:teuthology.orchestra.run.smithi131.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T00:55:28.817 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T00:55:28.817 INFO:teuthology.orchestra.run.smithi131.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-23T00:55:28.817 INFO:teuthology.orchestra.run.smithi131.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T00:55:28.817 INFO:teuthology.orchestra.run.smithi131.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T00:55:28.817 INFO:teuthology.orchestra.run.smithi131.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T00:55:28.817 INFO:teuthology.orchestra.run.smithi131.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T00:55:28.818 INFO:teuthology.orchestra.run.smithi131.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T00:55:28.818 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T00:55:28.818 INFO:teuthology.orchestra.run.smithi131.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T00:55:28.818 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T00:55:28.818 INFO:teuthology.orchestra.run.smithi131.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T00:55:28.818 INFO:teuthology.orchestra.run.smithi131.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T00:55:28.818 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.818 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.819 INFO:teuthology.orchestra.run.smithi131.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:55:28.819 INFO:teuthology.orchestra.run.smithi131.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-23T00:55:28.819 INFO:teuthology.orchestra.run.smithi131.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T00:55:28.819 INFO:teuthology.orchestra.run.smithi131.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-23T00:55:28.819 INFO:teuthology.orchestra.run.smithi131.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T00:55:28.820 INFO:teuthology.orchestra.run.smithi131.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-23T00:55:28.820 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T00:55:28.820 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2023-06-23T00:55:28.986 DEBUG:teuthology.parallel:result is None 2023-06-23T00:56:13.485 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-06-23T00:56:13.486 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-06-23T00:56:13.486 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-06-23T00:56:14.042 INFO:teuthology.orchestra.run.smithi046.stdout:(39/120): liboath-2.6.2-3.el8.x86_64.rpm 457 B/s | 59 kB 02:11 2023-06-23T00:56:14.050 INFO:teuthology.orchestra.run.smithi046.stdout:(40/120): libunwind-1.3.1-3.el8.x86_64.rpm 585 B/s | 75 kB 02:11 2023-06-23T00:56:14.059 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm (IP: 8.43.85.72) 2023-06-23T00:56:14.065 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:56:14.304 INFO:teuthology.orchestra.run.smithi046.stdout:(41/120): gperftools-libs-2.7-9.el8.x86_64.rpm 2.3 kB/s | 306 kB 02:12 2023-06-23T00:58:24.557 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-06-23T00:58:24.558 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/p/python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-06-23T00:58:24.558 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] python3-beautifulsoup4-4.6.3-2.el8.1.noarch.rpm: Curl error (7): Couldn't connect to server for http://mirror.es.its.nyu.edu/epel/8/Everything/x86_64/Packages/p/python3-beautifulsoup4-4.6.3-2.el8.1.noarch.rpm [Failed to connect to mirror.es.its.nyu.edu port 80: Connection timed out] 2023-06-23T00:58:24.566 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm (IP: 8.43.85.72) 2023-06-23T00:58:24.567 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] python3-beautifulsoup4-4.6.3-2.el8.1.noarch.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-beautifulsoup4-4.6.3-2.el8.1.noarch.rpm (IP: 8.43.85.72) 2023-06-23T00:58:24.567 INFO:teuthology.orchestra.run.smithi046.stdout:[MIRROR] python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T00:58:25.121 INFO:teuthology.orchestra.run.smithi046.stdout:(42/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 340 B/s | 44 kB 02:11 2023-06-23T00:58:25.207 INFO:teuthology.orchestra.run.smithi046.stdout:(43/120): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 kB/s | 185 kB 02:10 2023-06-23T00:58:25.211 INFO:teuthology.orchestra.run.smithi046.stdout:(44/120): python3-cachetools-3.1.1-4.el8.noarch 368 kB/s | 33 kB 00:00 2023-06-23T00:58:25.297 INFO:teuthology.orchestra.run.smithi046.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 182 kB/s | 16 kB 00:00 2023-06-23T00:58:25.377 INFO:teuthology.orchestra.run.smithi046.stdout:(46/120): python3-asyncssh-2.7.0-2.el8.noarch.r 3.6 kB/s | 476 kB 02:11 2023-06-23T00:58:25.468 INFO:teuthology.orchestra.run.smithi046.stdout:(47/120): python3-google-auth-1.1.1-10.el8.noar 901 kB/s | 81 kB 00:00 2023-06-23T00:58:25.479 INFO:teuthology.orchestra.run.smithi046.stdout:(48/120): python3-cherrypy-18.4.0-1.el8.noarch. 2.1 MB/s | 384 kB 00:00 2023-06-23T00:58:25.479 INFO:teuthology.orchestra.run.smithi046.stdout:(49/120): python3-cheroot-8.5.2-1.el8.noarch.rp 643 kB/s | 173 kB 00:00 2023-06-23T00:58:25.557 INFO:teuthology.orchestra.run.smithi046.stdout:(50/120): python3-jaraco-6.2-6.el8.noarch.rpm 124 kB/s | 11 kB 00:00 2023-06-23T00:58:25.569 INFO:teuthology.orchestra.run.smithi046.stdout:(51/120): python3-jaraco-functools-2.0-4.el8.no 196 kB/s | 18 kB 00:00 2023-06-23T00:58:25.647 INFO:teuthology.orchestra.run.smithi046.stdout:(52/120): python3-logutils-0.3.5-11.el8.noarch. 543 kB/s | 49 kB 00:00 2023-06-23T00:58:25.659 INFO:teuthology.orchestra.run.smithi046.stdout:(53/120): python3-more-itertools-7.2.0-3.el8.no 653 kB/s | 59 kB 00:00 2023-06-23T00:58:25.738 INFO:teuthology.orchestra.run.smithi046.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 670 kB/s | 60 kB 00:00 2023-06-23T00:58:25.754 INFO:teuthology.orchestra.run.smithi046.stdout:(55/120): python3-pecan-1.3.2-9.el8.noarch.rpm 2.9 MB/s | 283 kB 00:00 2023-06-23T00:58:25.828 INFO:teuthology.orchestra.run.smithi046.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 174 kB/s | 16 kB 00:00 2023-06-23T00:58:25.845 INFO:teuthology.orchestra.run.smithi046.stdout:(57/120): python3-kubernetes-11.0.0-6.el8.noarc 3.2 MB/s | 1.2 MB 00:00 2023-06-23T00:58:25.846 INFO:teuthology.orchestra.run.smithi046.stdout:(58/120): python3-repoze-lru-0.7-6.el8.noarch.r 375 kB/s | 34 kB 00:00 2023-06-23T00:58:25.919 INFO:teuthology.orchestra.run.smithi046.stdout:(59/120): python3-routes-2.4.1-12.el8.noarch.rp 2.1 MB/s | 196 kB 00:00 2023-06-23T00:58:25.933 INFO:teuthology.orchestra.run.smithi046.stdout:(60/120): python3-rsa-4.8-1.el8.noarch.rpm 683 kB/s | 61 kB 00:00 2023-06-23T00:58:25.934 INFO:teuthology.orchestra.run.smithi046.stdout:(61/120): python3-simplegeneric-0.8.1-17.el8.no 208 kB/s | 19 kB 00:00 2023-06-23T00:58:26.009 INFO:teuthology.orchestra.run.smithi046.stdout:(62/120): python3-singledispatch-3.4.0.3-18.el8 267 kB/s | 24 kB 00:00 2023-06-23T00:58:26.023 INFO:teuthology.orchestra.run.smithi046.stdout:(63/120): python3-tempora-1.14.1-5.el8.noarch.r 326 kB/s | 29 kB 00:00 2023-06-23T00:58:26.024 INFO:teuthology.orchestra.run.smithi046.stdout:(64/120): python3-trustme-0.6.0-4.el8.noarch.rp 307 kB/s | 27 kB 00:00 2023-06-23T00:58:26.101 INFO:teuthology.orchestra.run.smithi046.stdout:(65/120): python3-waitress-1.4.3-1.el8.noarch.r 2.7 MB/s | 255 kB 00:00 2023-06-23T00:58:26.116 INFO:teuthology.orchestra.run.smithi046.stdout:(66/120): python3-webob-1.8.5-1.el8.1.noarch.rp 2.7 MB/s | 251 kB 00:00 2023-06-23T00:58:26.117 INFO:teuthology.orchestra.run.smithi046.stdout:(67/120): python3-websocket-client-0.56.0-5.el8 661 kB/s | 61 kB 00:00 2023-06-23T00:58:26.191 INFO:teuthology.orchestra.run.smithi046.stdout:(68/120): python3-webtest-2.0.33-1.el8.noarch.r 947 kB/s | 85 kB 00:00 2023-06-23T00:58:26.197 INFO:teuthology.orchestra.run.smithi046.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 13 MB/s | 70 kB 00:00 2023-06-23T00:58:26.215 INFO:teuthology.orchestra.run.smithi046.stdout:(70/120): python3-zc-lockfile-2.0-2.el8.noarch. 233 kB/s | 23 kB 00:00 2023-06-23T00:58:26.246 INFO:teuthology.orchestra.run.smithi046.stdout:(71/120): python3-prettytable-0.7.2-14.el8.noar 1.4 MB/s | 44 kB 00:00 2023-06-23T00:58:26.260 INFO:teuthology.orchestra.run.smithi046.stdout:(72/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.8 MB/s | 27 kB 00:00 2023-06-23T00:58:26.277 INFO:teuthology.orchestra.run.smithi046.stdout:(73/120): jq-1.5-12.el8.x86_64.rpm 9.7 MB/s | 161 kB 00:00 2023-06-23T00:58:26.296 INFO:teuthology.orchestra.run.smithi046.stdout:(74/120): flex-2.6.1-9.el8.x86_64.rpm 16 MB/s | 320 kB 00:00 2023-06-23T00:58:26.318 INFO:teuthology.orchestra.run.smithi046.stdout:(75/120): python3-werkzeug-0.12.2-4.el8.noarch. 21 MB/s | 457 kB 00:00 2023-06-23T00:58:26.392 INFO:teuthology.orchestra.run.smithi046.stdout:(76/120): thrift-0.13.0-2.el8.x86_64.rpm 6.4 MB/s | 1.7 MB 00:00 2023-06-23T00:58:26.427 INFO:teuthology.orchestra.run.smithi046.stdout:(77/120): python3-babel-2.5.1-5.el8.noarch.rpm 44 MB/s | 4.8 MB 00:00 2023-06-23T00:58:26.428 INFO:teuthology.orchestra.run.smithi046.stdout:(78/120): python3-markupsafe-0.23-19.el8.x86_64 1.0 MB/s | 39 kB 00:00 2023-06-23T00:58:26.450 INFO:teuthology.orchestra.run.smithi046.stdout:(79/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 12 MB/s | 259 kB 00:00 2023-06-23T00:58:26.565 INFO:teuthology.orchestra.run.smithi046.stdout:(80/120): python3-scipy-1.0.0-20.module+el8.1.0 40 MB/s | 15 MB 00:00 2023-06-23T00:58:26.567 INFO:teuthology.orchestra.run.smithi046.stdout:(81/120): bison-3.0.4-10.el8.x86_64.rpm 4.8 MB/s | 688 kB 00:00 2023-06-23T00:58:26.570 INFO:teuthology.orchestra.run.smithi046.stdout:(82/120): python3-numpy-f2py-1.14.3-9.el8.x86_6 1.8 MB/s | 225 kB 00:00 2023-06-23T00:58:26.584 INFO:teuthology.orchestra.run.smithi046.stdout:(83/120): python3-pyasn1-0.3.7-6.el8.noarch.rpm 6.4 MB/s | 126 kB 00:00 2023-06-23T00:58:26.595 INFO:teuthology.orchestra.run.smithi046.stdout:(84/120): oniguruma-6.8.2-2.el8.x86_64.rpm 7.5 MB/s | 187 kB 00:00 2023-06-23T00:58:26.609 INFO:teuthology.orchestra.run.smithi046.stdout:(85/120): python3-jinja2-2.10.1-2.el8_0.noarch. 21 MB/s | 538 kB 00:00 2023-06-23T00:58:26.614 INFO:teuthology.orchestra.run.smithi046.stdout:(86/120): socat-1.7.3.3-2.el8.x86_64.rpm 15 MB/s | 302 kB 00:00 2023-06-23T00:58:26.628 INFO:teuthology.orchestra.run.smithi046.stdout:(87/120): python3-mako-1.0.6-13.el8.noarch.rpm 8.5 MB/s | 157 kB 00:00 2023-06-23T00:58:26.629 INFO:teuthology.orchestra.run.smithi046.stdout:(88/120): python3-pyasn1-modules-0.3.7-6.el8.no 7.3 MB/s | 110 kB 00:00 2023-06-23T00:58:26.642 INFO:teuthology.orchestra.run.smithi046.stdout:(89/120): python3-numpy-1.14.3-9.el8.x86_64.rpm 50 MB/s | 3.7 MB 00:00 2023-06-23T00:58:26.643 INFO:teuthology.orchestra.run.smithi046.stdout:(90/120): python3-jsonpointer-1.10-11.el8.noarc 1.4 MB/s | 20 kB 00:00 2023-06-23T00:58:26.647 INFO:teuthology.orchestra.run.smithi046.stdout:(91/120): librdkafka-0.11.4-1.el8.x86_64.rpm 18 MB/s | 353 kB 00:00 2023-06-23T00:58:26.660 INFO:teuthology.orchestra.run.smithi046.stdout:(92/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.4 MB/s | 54 kB 00:00 2023-06-23T00:58:26.667 INFO:teuthology.orchestra.run.smithi046.stdout:(93/120): libtool-2.4.6-25.el8.x86_64.rpm 28 MB/s | 709 kB 00:00 2023-06-23T00:58:26.693 INFO:teuthology.orchestra.run.smithi046.stdout:(94/120): automake-1.16.1-7.el8.noarch.rpm 27 MB/s | 713 kB 00:00 2023-06-23T00:58:26.708 INFO:teuthology.orchestra.run.smithi046.stdout:(95/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 1.3 MB/s | 19 kB 00:00 2023-06-23T00:58:26.721 INFO:teuthology.orchestra.run.smithi046.stdout:(96/120): python3-jwt-1.6.1-2.el8.noarch.rpm 3.2 MB/s | 43 kB 00:00 2023-06-23T00:58:26.746 INFO:teuthology.orchestra.run.smithi046.stdout:(97/120): openblas-threads-0.3.12-1.el8.x86_64. 48 MB/s | 4.7 MB 00:00 2023-06-23T00:58:26.749 INFO:teuthology.orchestra.run.smithi046.stdout:(98/120): python3-oauthlib-2.1.0-1.el8.noarch.r 5.5 MB/s | 155 kB 00:00 2023-06-23T00:58:26.766 INFO:teuthology.orchestra.run.smithi046.stdout:(99/120): openblas-0.3.12-1.el8.x86_64.rpm 43 MB/s | 4.6 MB 00:00 2023-06-23T00:58:26.767 INFO:teuthology.orchestra.run.smithi046.stdout:(100/120): keyutils-libs-devel-1.5.10-6.el8.x86 2.6 MB/s | 48 kB 00:00 2023-06-23T00:58:26.770 INFO:teuthology.orchestra.run.smithi046.stdout:(101/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 9.1 MB/s | 220 kB 00:00 2023-06-23T00:58:26.782 INFO:teuthology.orchestra.run.smithi046.stdout:(102/120): python3-requests-oauthlib-1.0.0-1.el 2.9 MB/s | 43 kB 00:00 2023-06-23T00:58:26.785 INFO:teuthology.orchestra.run.smithi046.stdout:(103/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 12 MB/s | 229 kB 00:00 2023-06-23T00:58:26.797 INFO:teuthology.orchestra.run.smithi046.stdout:(104/120): libverto-devel-0.3.0-5.el8.x86_64.rp 1.2 MB/s | 18 kB 00:00 2023-06-23T00:58:26.809 INFO:teuthology.orchestra.run.smithi046.stdout:(105/120): pcre2-devel-10.32-2.el8.x86_64.rpm 24 MB/s | 605 kB 00:00 2023-06-23T00:58:26.813 INFO:teuthology.orchestra.run.smithi046.stdout:(106/120): fuse-2.9.7-12.el8.x86_64.rpm 5.2 MB/s | 83 kB 00:00 2023-06-23T00:58:26.825 INFO:teuthology.orchestra.run.smithi046.stdout:(107/120): openssl-devel-1.1.1g-15.el8_3.x86_64 42 MB/s | 2.3 MB 00:00 2023-06-23T00:58:26.827 INFO:teuthology.orchestra.run.smithi046.stdout:(108/120): elfutils-libelf-devel-0.182-3.el8.x8 3.3 MB/s | 59 kB 00:00 2023-06-23T00:58:26.837 INFO:teuthology.orchestra.run.smithi046.stdout:(109/120): libquadmath-8.4.1-1.el8.x86_64.rpm 6.8 MB/s | 169 kB 00:00 2023-06-23T00:58:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:(110/120): libcom_err-devel-1.45.6-1.el8.x86_64 2.3 MB/s | 38 kB 00:00 2023-06-23T00:58:26.844 INFO:teuthology.orchestra.run.smithi046.stdout:(111/120): libsepol-devel-2.9-2.el8.x86_64.rpm 4.9 MB/s | 86 kB 00:00 2023-06-23T00:58:26.858 INFO:teuthology.orchestra.run.smithi046.stdout:(112/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.7 MB/s | 58 kB 00:00 2023-06-23T00:58:26.868 INFO:teuthology.orchestra.run.smithi046.stdout:(113/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 23 MB/s | 559 kB 00:00 2023-06-23T00:58:26.868 INFO:teuthology.orchestra.run.smithi046.stdout:(114/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.7 MB/s | 47 kB 00:00 2023-06-23T00:58:26.883 INFO:teuthology.orchestra.run.smithi046.stdout:(115/120): libgfortran-8.4.1-1.el8.x86_64.rpm 26 MB/s | 639 kB 00:00 2023-06-23T00:58:26.883 INFO:teuthology.orchestra.run.smithi046.stdout:(116/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 4.5 MB/s | 70 kB 00:00 2023-06-23T00:58:26.887 INFO:teuthology.orchestra.run.smithi046.stdout:(117/120): libselinux-devel-2.9-5.el8.x86_64.rp 11 MB/s | 200 kB 00:00 2023-06-23T00:58:26.898 INFO:teuthology.orchestra.run.smithi046.stdout:(118/120): libuuid-devel-2.32.1-27.el8.x86_64.r 5.9 MB/s | 97 kB 00:00 2023-06-23T00:58:27.208 INFO:teuthology.orchestra.run.smithi046.stdout:(119/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 11 MB/s | 3.6 MB 00:00 2023-06-23T00:58:27.667 INFO:teuthology.orchestra.run.smithi046.stdout:(120/120): librados2-17.2.6-719.g42ddc46e.el8.x 4.4 MB/s | 3.5 MB 00:00 2023-06-23T00:58:27.673 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2023-06-23T00:58:27.673 INFO:teuthology.orchestra.run.smithi046.stdout:Total 742 kB/s | 201 MB 04:37 2023-06-23T00:58:27.728 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2023-06-23T00:58:27.851 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2023-06-23T00:58:27.852 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2023-06-23T00:58:29.317 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2023-06-23T00:58:29.322 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2023-06-23T00:58:34.342 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2023-06-23T00:58:34.611 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-23T00:58:34.817 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/122 2023-06-23T00:58:35.030 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T00:58:35.180 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T00:58:35.447 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T00:58:35.842 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T00:58:36.039 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T00:58:36.490 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T00:58:36.705 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:58:36.839 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:58:37.030 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:58:37.131 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:58:37.503 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T00:58:38.222 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-23T00:58:38.380 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:58:38.524 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:58:38.768 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-23T00:58:38.988 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-23T00:58:39.146 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:58:39.290 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:58:39.472 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T00:58:39.540 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T00:58:39.668 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 15/122 2023-06-23T00:58:39.814 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T00:58:39.957 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T00:58:40.096 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T00:58:40.707 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T00:58:40.880 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T00:58:41.597 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T00:58:41.813 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T00:58:42.574 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T00:58:42.859 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 20/122 2023-06-23T00:58:43.071 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 21/122 2023-06-23T00:58:43.218 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 22/122 2023-06-23T00:58:43.499 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/122 2023-06-23T00:58:43.819 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/122 2023-06-23T00:58:44.205 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-23T00:58:44.395 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-23T00:58:44.517 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-23T00:58:44.775 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T00:58:45.827 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/122 2023-06-23T00:58:46.165 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/122 2023-06-23T00:58:46.458 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/122 2023-06-23T00:58:46.720 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 32/122 2023-06-23T00:58:46.967 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 33/122 2023-06-23T00:58:47.245 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T00:58:49.420 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T00:58:49.743 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 35/122 2023-06-23T00:58:49.944 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-23T00:58:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-23T00:58:50.352 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-23T00:58:50.559 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-23T00:58:50.781 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-23T00:58:50.930 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 41/122 2023-06-23T00:58:51.125 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 42/122 2023-06-23T00:58:51.314 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 43/122 2023-06-23T00:58:51.448 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.7-12.el8.x86_64 44/122 2023-06-23T00:58:51.610 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 45/122 2023-06-23T00:58:51.812 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 46/122 2023-06-23T00:58:51.950 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/122 2023-06-23T00:58:52.307 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/122 2023-06-23T00:58:52.566 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 49/122 2023-06-23T00:58:52.863 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 50/122 2023-06-23T00:58:53.216 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 51/122 2023-06-23T00:58:53.620 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : automake-1.16.1-7.el8.noarch 52/122 2023-06-23T00:58:53.812 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/122 2023-06-23T00:58:53.986 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/122 2023-06-23T00:58:54.200 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 55/122 2023-06-23T00:58:54.386 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T00:58:54.468 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T00:58:54.663 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : jq-1.5-12.el8.x86_64 57/122 2023-06-23T00:58:54.891 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 58/122 2023-06-23T00:58:55.047 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-23T00:58:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-23T00:58:55.547 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-23T00:58:55.772 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-23T00:58:55.967 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-23T00:58:56.117 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-23T00:58:56.284 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-23T00:58:56.495 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-23T00:58:56.712 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-23T00:58:56.912 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-23T00:58:57.090 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-23T00:58:57.269 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-23T00:58:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-23T00:58:57.675 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-23T00:58:57.894 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-23T00:58:58.223 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-23T00:58:58.486 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-23T00:58:58.663 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-23T00:58:58.934 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-23T00:59:00.154 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-23T00:59:00.444 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-23T00:59:00.667 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-23T00:59:00.994 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-23T00:59:01.262 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-23T00:59:01.600 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-23T00:59:01.868 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-23T00:59:02.055 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-23T00:59:02.286 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-23T00:59:07.920 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:59:08.152 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:59:08.776 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T00:59:09.049 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T00:59:09.164 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T00:59:09.335 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T00:59:27.917 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T00:59:27.917 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2023-06-23T00:59:27.917 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2023-06-23T00:59:27.917 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /dev 2023-06-23T00:59:27.917 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /run 2023-06-23T00:59:27.918 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2023-06-23T00:59:27.918 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2023-06-23T00:59:27.919 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2023-06-23T00:59:27.920 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2023-06-23T00:59:27.920 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /dev 2023-06-23T00:59:27.920 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:59:28.324 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T00:59:29.926 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T00:59:30.080 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T00:59:30.316 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T00:59:30.491 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T00:59:30.904 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T00:59:31.040 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T00:59:31.059 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T00:59:34.159 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T00:59:34.331 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T00:59:34.472 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-23T00:59:34.680 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-23T00:59:34.864 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T00:59:35.092 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T00:59:35.337 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T00:59:36.371 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T00:59:36.581 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T00:59:36.684 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T00:59:37.052 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-23T00:59:37.229 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T00:59:37.267 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T00:59:37.447 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T00:59:37.459 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T00:59:39.197 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-23T00:59:39.412 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T00:59:39.508 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T00:59:39.698 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T00:59:39.748 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T00:59:39.956 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T00:59:40.326 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T00:59:40.506 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T00:59:45.801 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T00:59:46.270 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-23T00:59:46.463 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T00:59:47.222 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T00:59:47.600 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 110/122 2023-06-23T00:59:47.776 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 111/122 2023-06-23T00:59:48.125 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 112/122 2023-06-23T00:59:48.252 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-23T00:59:48.428 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-23T00:59:48.572 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-23T00:59:48.773 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-23T00:59:48.940 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-23T00:59:49.150 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 118/122 2023-06-23T00:59:49.321 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T00:59:49.405 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T00:59:49.571 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T00:59:49.581 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T00:59:49.581 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T00:59:49.783 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T00:59:49.783 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:59:50.317 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:59:50.317 INFO:teuthology.orchestra.run.smithi046.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-23T00:59:50.318 INFO:teuthology.orchestra.run.smithi046.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-23T00:59:50.318 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:59:50.671 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-23T00:59:50.671 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-23T00:59:50.672 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-23T00:59:50.672 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T00:59:50.672 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-23T00:59:50.672 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T00:59:50.672 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T00:59:50.672 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T00:59:50.673 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-23T00:59:50.673 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T00:59:50.673 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-23T00:59:50.673 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-23T00:59:50.673 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T00:59:50.673 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-23T00:59:50.673 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-23T00:59:50.674 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-23T00:59:50.674 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-23T00:59:50.674 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-23T00:59:50.675 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-23T00:59:50.675 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-23T00:59:50.675 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-23T00:59:50.676 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-23T00:59:50.676 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-23T00:59:50.676 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-23T00:59:50.676 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-23T00:59:50.687 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-23T00:59:50.687 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-23T00:59:50.687 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T00:59:50.688 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-23T00:59:50.688 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-23T00:59:50.688 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-23T00:59:50.688 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-23T00:59:50.688 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-23T00:59:50.688 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-23T00:59:50.689 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-23T00:59:50.689 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-23T00:59:50.689 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-23T00:59:50.689 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-23T00:59:50.689 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-23T00:59:50.689 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-23T00:59:50.689 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-23T00:59:50.689 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-23T00:59:50.690 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-23T00:59:50.690 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-23T00:59:50.690 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-23T00:59:50.690 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-23T00:59:50.690 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-23T00:59:50.690 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-23T00:59:50.690 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-23T00:59:50.691 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-23T00:59:50.691 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-23T00:59:50.691 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-23T00:59:50.691 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-23T00:59:50.691 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-23T00:59:50.691 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-23T00:59:50.691 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-23T00:59:50.691 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-23T00:59:50.692 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-23T00:59:50.692 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-23T00:59:50.692 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-23T00:59:50.692 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-23T00:59:50.692 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-23T00:59:50.692 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-23T00:59:50.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-23T00:59:50.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-23T00:59:50.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-23T00:59:50.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-23T00:59:50.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-23T00:59:50.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-23T00:59:50.693 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-23T00:59:50.694 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-23T00:59:50.694 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 72/122 2023-06-23T00:59:50.694 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/122 2023-06-23T00:59:50.694 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/122 2023-06-23T00:59:50.694 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : jq-1.5-12.el8.x86_64 75/122 2023-06-23T00:59:50.694 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flex-2.6.1-9.el8.x86_64 76/122 2023-06-23T00:59:50.694 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/122 2023-06-23T00:59:50.694 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 78/122 2023-06-23T00:59:50.695 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 79/122 2023-06-23T00:59:50.695 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/122 2023-06-23T00:59:50.695 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 81/122 2023-06-23T00:59:50.695 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 82/122 2023-06-23T00:59:50.695 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 83/122 2023-06-23T00:59:50.696 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 84/122 2023-06-23T00:59:50.696 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/122 2023-06-23T00:59:50.696 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 86/122 2023-06-23T00:59:50.696 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 87/122 2023-06-23T00:59:50.696 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 88/122 2023-06-23T00:59:50.697 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 89/122 2023-06-23T00:59:50.697 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 90/122 2023-06-23T00:59:50.697 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 91/122 2023-06-23T00:59:50.697 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 92/122 2023-06-23T00:59:50.697 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 93/122 2023-06-23T00:59:50.697 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 94/122 2023-06-23T00:59:50.697 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/122 2023-06-23T00:59:50.698 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : automake-1.16.1-7.el8.noarch 96/122 2023-06-23T00:59:50.698 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 97/122 2023-06-23T00:59:50.698 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/122 2023-06-23T00:59:50.698 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/122 2023-06-23T00:59:50.698 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 100/122 2023-06-23T00:59:50.698 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 101/122 2023-06-23T00:59:50.698 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 102/122 2023-06-23T00:59:50.699 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/122 2023-06-23T00:59:50.699 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 104/122 2023-06-23T00:59:50.699 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/122 2023-06-23T00:59:50.699 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 106/122 2023-06-23T00:59:50.699 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 107/122 2023-06-23T00:59:50.700 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 108/122 2023-06-23T00:59:50.700 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 109/122 2023-06-23T00:59:50.700 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 110/122 2023-06-23T00:59:50.700 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 111/122 2023-06-23T00:59:50.700 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 112/122 2023-06-23T00:59:50.701 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 113/122 2023-06-23T00:59:50.701 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 114/122 2023-06-23T00:59:50.701 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 115/122 2023-06-23T00:59:50.701 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 116/122 2023-06-23T00:59:50.701 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 117/122 2023-06-23T00:59:50.701 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 118/122 2023-06-23T00:59:50.702 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-23T00:59:50.702 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-23T00:59:50.702 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-23T00:59:53.049 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T00:59:53.049 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:06:13 ago on Fri 23 Jun 2023 12:53:40 AM UTC. 2023-06-23T00:59:55.052 INFO:teuthology.orchestra.run.smithi046.stdout:Installed products updated. 2023-06-23T00:59:55.184 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:59:55.185 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2023-06-23T00:59:55.185 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.185 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.185 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2023-06-23T00:59:55.185 INFO:teuthology.orchestra.run.smithi046.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T00:59:55.185 INFO:teuthology.orchestra.run.smithi046.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T00:59:55.185 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.186 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.186 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.186 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.186 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:59:55.186 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.186 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.186 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.187 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:59:55.187 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:59:55.187 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:59:55.187 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:59:55.187 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:59:55.187 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.188 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.188 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:59:55.188 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.188 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.188 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.188 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:59:55.188 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:59:55.189 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.189 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T00:59:55.189 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-23T00:59:55.189 INFO:teuthology.orchestra.run.smithi046.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T00:59:55.189 INFO:teuthology.orchestra.run.smithi046.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-23T00:59:55.189 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T00:59:55.189 INFO:teuthology.orchestra.run.smithi046.stdout: jq-1.5-12.el8.x86_64 2023-06-23T00:59:55.190 INFO:teuthology.orchestra.run.smithi046.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-23T00:59:55.190 INFO:teuthology.orchestra.run.smithi046.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-23T00:59:55.190 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T00:59:55.190 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.190 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.191 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.191 INFO:teuthology.orchestra.run.smithi046.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-23T00:59:55.191 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-23T00:59:55.191 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T00:59:55.191 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-23T00:59:55.192 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T00:59:55.192 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.192 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.192 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-23T00:59:55.192 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.192 INFO:teuthology.orchestra.run.smithi046.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-23T00:59:55.192 INFO:teuthology.orchestra.run.smithi046.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-23T00:59:55.192 INFO:teuthology.orchestra.run.smithi046.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T00:59:55.193 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T00:59:55.193 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-23T00:59:55.193 INFO:teuthology.orchestra.run.smithi046.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-23T00:59:55.193 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T00:59:55.193 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T00:59:55.193 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-23T00:59:55.193 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-23T00:59:55.194 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-23T00:59:55.194 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-23T00:59:55.194 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-23T00:59:55.194 INFO:teuthology.orchestra.run.smithi046.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-23T00:59:55.194 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T00:59:55.194 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-23T00:59:55.194 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T00:59:55.194 INFO:teuthology.orchestra.run.smithi046.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T00:59:55.195 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T00:59:55.195 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.195 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.195 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.195 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T00:59:55.195 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T00:59:55.196 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T00:59:55.196 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T00:59:55.196 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T00:59:55.196 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T00:59:55.196 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-23T00:59:55.196 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T00:59:55.196 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T00:59:55.196 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T00:59:55.197 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T00:59:55.197 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T00:59:55.197 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-23T00:59:55.197 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T00:59:55.197 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T00:59:55.197 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T00:59:55.197 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-23T00:59:55.198 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-23T00:59:55.198 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T00:59:55.198 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T00:59:55.198 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T00:59:55.198 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T00:59:55.198 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T00:59:55.198 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T00:59:55.198 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T00:59:55.199 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.199 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.199 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T00:59:55.199 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T00:59:55.199 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.199 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T00:59:55.200 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T00:59:55.200 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-23T00:59:55.200 INFO:teuthology.orchestra.run.smithi046.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T00:59:55.200 INFO:teuthology.orchestra.run.smithi046.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T00:59:55.200 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T00:59:55.200 INFO:teuthology.orchestra.run.smithi046.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T00:59:55.200 INFO:teuthology.orchestra.run.smithi046.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T00:59:55.201 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T00:59:55.201 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T00:59:55.201 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T00:59:55.201 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T00:59:55.201 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T00:59:55.201 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.201 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.201 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T00:59:55.202 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-23T00:59:55.202 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T00:59:55.202 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-23T00:59:55.202 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T00:59:55.203 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-23T00:59:55.203 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T00:59:55.203 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2023-06-23T00:59:55.395 DEBUG:teuthology.parallel:result is None 2023-06-23T00:59:55.396 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:59:55.541 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T00:59:55.563 INFO:teuthology.orchestra.run.smithi046.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T00:59:55.564 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T00:59:55.565 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T00:59:55.565 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:59:55.773 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T00:59:55.798 INFO:teuthology.orchestra.run.smithi092.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T00:59:55.798 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T00:59:55.799 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T00:59:55.799 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:59:56.013 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T00:59:56.041 INFO:teuthology.orchestra.run.smithi131.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T00:59:56.041 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T00:59:56.042 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T00:59:56.042 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-06-23T00:59:56.043 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T00:59:56.044 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T00:59:56.086 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T00:59:56.086 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T00:59:56.128 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-06-23T00:59:56.129 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T00:59:56.172 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-06-23T00:59:56.173 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T00:59:56.173 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T00:59:56.199 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T00:59:56.265 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T00:59:56.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T00:59:56.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T00:59:56.358 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-06-23T00:59:56.359 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T00:59:56.386 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T00:59:56.451 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-06-23T00:59:56.452 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T00:59:56.452 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T00:59:56.478 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T00:59:56.544 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T00:59:56.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T00:59:56.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T00:59:56.634 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-06-23T00:59:56.635 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T00:59:56.660 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T00:59:56.725 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-06-23T00:59:56.725 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T00:59:56.726 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T00:59:56.751 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T00:59:56.817 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T00:59:56.817 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T00:59:56.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T00:59:56.907 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-06-23T00:59:56.908 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T00:59:56.932 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T00:59:56.997 INFO:teuthology.run_tasks:Running task cephadm... 2023-06-23T00:59:57.086 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'SLOW_OPS', 'slow request'], 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'} 2023-06-23T00:59:57.087 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:59:57.099 INFO:tasks.cephadm:Cluster fsid is 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T00:59:57.100 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-06-23T00:59:57.100 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.46', 'mon.b': '172.21.15.92', 'mon.c': '172.21.15.131'} 2023-06-23T00:59:57.101 INFO:tasks.cephadm:First mon is mon.a on smithi046 2023-06-23T00:59:57.101 INFO:tasks.cephadm:First mgr is x 2023-06-23T00:59:57.101 INFO:tasks.cephadm:Normalizing hostnames... 2023-06-23T00:59:57.102 DEBUG:teuthology.orchestra.run.smithi046:> sudo hostname $(hostname -s) 2023-06-23T00:59:57.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo hostname $(hostname -s) 2023-06-23T00:59:57.154 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname $(hostname -s) 2023-06-23T00:59:57.180 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 42ddc46e6e6582d8e29a5a982a81886fe028be2c)... 2023-06-23T00:59:57.181 DEBUG:teuthology.orchestra.run.smithi046:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/42ddc46e6e6582d8e29a5a982a81886fe028be2c/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-06-23T00:59:57.414 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-rw-r--. 1 ubuntu ubuntu 366215 Jun 23 00:59 /home/ubuntu/cephtest/cephadm 2023-06-23T00:59:57.415 DEBUG:teuthology.orchestra.run.smithi092:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/42ddc46e6e6582d8e29a5a982a81886fe028be2c/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-06-23T00:59:57.623 INFO:teuthology.orchestra.run.smithi092.stdout:-rw-rw-r--. 1 ubuntu ubuntu 366215 Jun 23 00:59 /home/ubuntu/cephtest/cephadm 2023-06-23T00:59:57.623 DEBUG:teuthology.orchestra.run.smithi131:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/42ddc46e6e6582d8e29a5a982a81886fe028be2c/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-06-23T00:59:57.763 INFO:teuthology.orchestra.run.smithi131.stdout:-rw-rw-r--. 1 ubuntu ubuntu 366215 Jun 23 00:59 /home/ubuntu/cephtest/cephadm 2023-06-23T00:59:57.765 DEBUG:teuthology.orchestra.run.smithi046:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-06-23T00:59:57.784 DEBUG:teuthology.orchestra.run.smithi092:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-06-23T00:59:57.800 DEBUG:teuthology.orchestra.run.smithi131:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-06-23T00:59:57.833 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c on all hosts... 2023-06-23T00:59:57.833 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c pull 2023-06-23T00:59:57.835 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c pull 2023-06-23T00:59:57.843 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c pull 2023-06-23T00:59:59.398 INFO:teuthology.orchestra.run.smithi131.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c... 2023-06-23T00:59:59.430 INFO:teuthology.orchestra.run.smithi092.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c... 2023-06-23T01:00:03.786 INFO:teuthology.orchestra.run.smithi046.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c... 2023-06-23T01:00:36.509 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2023-06-23T01:00:36.510 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_version": "ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable)", 2023-06-23T01:00:36.510 INFO:teuthology.orchestra.run.smithi092.stdout: "image_id": "f3276efde470d44ef0ff8e259464d1d34a5015beb9d7c1215520bcf3b48fd710", 2023-06-23T01:00:36.510 INFO:teuthology.orchestra.run.smithi092.stdout: "repo_digests": [ 2023-06-23T01:00:36.510 INFO:teuthology.orchestra.run.smithi092.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b9658f1f8fdc87272ce6f577d5ecedf811fa77b9a03555eb6b1de17c25ce4236" 2023-06-23T01:00:36.510 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2023-06-23T01:00:36.511 INFO:teuthology.orchestra.run.smithi092.stdout:} 2023-06-23T01:00:38.467 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-06-23T01:00:38.467 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable)", 2023-06-23T01:00:38.468 INFO:teuthology.orchestra.run.smithi046.stdout: "image_id": "f3276efde470d44ef0ff8e259464d1d34a5015beb9d7c1215520bcf3b48fd710", 2023-06-23T01:00:38.468 INFO:teuthology.orchestra.run.smithi046.stdout: "repo_digests": [ 2023-06-23T01:00:38.468 INFO:teuthology.orchestra.run.smithi046.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b9658f1f8fdc87272ce6f577d5ecedf811fa77b9a03555eb6b1de17c25ce4236" 2023-06-23T01:00:38.468 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2023-06-23T01:00:38.469 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-06-23T01:00:49.180 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-06-23T01:00:49.181 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_version": "ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable)", 2023-06-23T01:00:49.181 INFO:teuthology.orchestra.run.smithi131.stdout: "image_id": "f3276efde470d44ef0ff8e259464d1d34a5015beb9d7c1215520bcf3b48fd710", 2023-06-23T01:00:49.181 INFO:teuthology.orchestra.run.smithi131.stdout: "repo_digests": [ 2023-06-23T01:00:49.182 INFO:teuthology.orchestra.run.smithi131.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b9658f1f8fdc87272ce6f577d5ecedf811fa77b9a03555eb6b1de17c25ce4236" 2023-06-23T01:00:49.182 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-06-23T01:00:49.182 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-06-23T01:00:49.200 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph 2023-06-23T01:00:49.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph 2023-06-23T01:00:49.269 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph 2023-06-23T01:00:49.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /etc/ceph 2023-06-23T01:00:49.326 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /etc/ceph 2023-06-23T01:00:49.353 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 777 /etc/ceph 2023-06-23T01:00:49.377 INFO:tasks.cephadm:Writing seed config... 2023-06-23T01:00:49.379 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-06-23T01:00:49.379 INFO:tasks.cephadm: override: [client] debug client = 20 2023-06-23T01:00:49.380 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-06-23T01:00:49.380 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-06-23T01:00:49.381 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-06-23T01:00:49.381 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-06-23T01:00:49.382 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-06-23T01:00:49.383 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-06-23T01:00:49.383 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-06-23T01:00:49.384 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-06-23T01:00:49.384 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-06-23T01:00:49.385 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-06-23T01:00:49.385 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-06-23T01:00:49.385 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-06-23T01:00:49.386 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-06-23T01:00:49.387 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-06-23T01:00:49.387 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-06-23T01:00:49.388 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-06-23T01:00:49.388 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-06-23T01:00:49.388 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-06-23T01:00:49.389 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-06-23T01:00:49.389 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-06-23T01:00:49.390 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-06-23T01:00:49.390 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-06-23T01:00:49.390 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-06-23T01:00:49.390 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-06-23T01:00:49.391 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-06-23T01:00:49.391 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-06-23T01:00:49.391 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-06-23T01:00:49.391 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-06-23T01:00:49.391 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-06-23T01:00:49.392 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-06-23T01:00:49.392 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2023-06-23T01:00:49.392 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-06-23T01:00:49.392 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-06-23T01:00:49.393 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-06-23T01:00:49.393 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-06-23T01:00:49.393 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2023-06-23T01:00:49.393 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-06-23T01:00:49.393 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-06-23T01:00:49.393 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-06-23T01:00:49.394 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-06-23T01:00:49.395 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T01:00:49.395 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-06-23T01:00:49.413 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 458642ee-1161-11ee-9b2b-001a4aab830c [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-06-23T01:00:49.414 DEBUG:teuthology.orchestra.run.smithi046:mon.a> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.a.service 2023-06-23T01:00:49.456 DEBUG:teuthology.orchestra.run.smithi046:mgr.x> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.x.service 2023-06-23T01:00:49.485 INFO:journalctl@ceph.mon.a.smithi046.stdout:-- Logs begin at Fri 2023-06-23 00:44:15 UTC. -- 2023-06-23T01:00:49.486 INFO:tasks.cephadm:Bootstrapping... 2023-06-23T01:00:49.487 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c -v bootstrap --fsid 458642ee-1161-11ee-9b2b-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.46 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-06-23T01:00:49.513 INFO:journalctl@ceph.mgr.x.smithi046.stdout:-- Logs begin at Fri 2023-06-23 00:44:15 UTC. -- 2023-06-23T01:00:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2023-06-23T01:00:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c', '-v', 'bootstrap', '--fsid', '458642ee-1161-11ee-9b2b-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.46', '--skip-admin-label'] 2023-06-23T01:00:49.847 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 3.2.3 2023-06-23T01:00:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-06-23T01:00:49.849 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying podman|docker is present... 2023-06-23T01:00:49.986 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 3.2.3 2023-06-23T01:00:49.987 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying lvm2 is present... 2023-06-23T01:00:49.987 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying time synchronization is in place... 2023-06-23T01:00:49.994 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-06-23T01:00:49.995 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-06-23T01:00:50.000 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-06-23T01:00:50.001 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2023-06-23T01:00:50.006 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout enabled 2023-06-23T01:00:50.012 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout active 2023-06-23T01:00:50.013 INFO:teuthology.orchestra.run.smithi046.stdout:Unit chronyd.service is enabled and running 2023-06-23T01:00:50.013 INFO:teuthology.orchestra.run.smithi046.stdout:Repeating the final host check... 2023-06-23T01:00:50.151 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 3.2.3 2023-06-23T01:00:50.152 INFO:teuthology.orchestra.run.smithi046.stdout:podman (/bin/podman) version 3.2.3 is present 2023-06-23T01:00:50.152 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl is present 2023-06-23T01:00:50.152 INFO:teuthology.orchestra.run.smithi046.stdout:lvcreate is present 2023-06-23T01:00:50.158 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-06-23T01:00:50.158 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-06-23T01:00:50.164 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-06-23T01:00:50.164 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2023-06-23T01:00:50.170 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout enabled 2023-06-23T01:00:50.176 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout active 2023-06-23T01:00:50.176 INFO:teuthology.orchestra.run.smithi046.stdout:Unit chronyd.service is enabled and running 2023-06-23T01:00:50.177 INFO:teuthology.orchestra.run.smithi046.stdout:Host looks OK 2023-06-23T01:00:50.177 INFO:teuthology.orchestra.run.smithi046.stdout:Cluster fsid: 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:00:50.177 INFO:teuthology.orchestra.run.smithi046.stdout:Acquiring lock 139846172645080 on /run/cephadm/458642ee-1161-11ee-9b2b-001a4aab830c.lock 2023-06-23T01:00:50.178 INFO:teuthology.orchestra.run.smithi046.stdout:Lock 139846172645080 acquired on /run/cephadm/458642ee-1161-11ee-9b2b-001a4aab830c.lock 2023-06-23T01:00:50.178 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying IP 172.21.15.46 port 3300 ... 2023-06-23T01:00:50.178 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying IP 172.21.15.46 port 6789 ... 2023-06-23T01:00:50.178 INFO:teuthology.orchestra.run.smithi046.stdout:Base mon IP(s) is [172.21.15.46:3300, 172.21.15.46:6789], mon addrv is [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2023-06-23T01:00:50.181 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-06-23T01:00:50.181 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.46 metric 100 2023-06-23T01:00:50.184 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-06-23T01:00:50.184 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-06-23T01:00:50.184 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1789sec hoplimit 64 pref medium 2023-06-23T01:00:50.186 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-06-23T01:00:50.186 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-06-23T01:00:50.186 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-06-23T01:00:50.187 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 3: ens1f0: mtu 1500 state UP qlen 1000 2023-06-23T01:00:50.187 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15f8/64 scope link 2023-06-23T01:00:50.187 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-06-23T01:00:50.188 INFO:teuthology.orchestra.run.smithi046.stdout:Mon IP `172.21.15.46` is in CIDR network `172.21.0.0/20` 2023-06-23T01:00:50.188 INFO:teuthology.orchestra.run.smithi046.stdout:Mon IP `172.21.15.46` is in CIDR network `172.21.0.0/20` 2023-06-23T01:00:50.188 INFO:teuthology.orchestra.run.smithi046.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-06-23T01:00:50.188 INFO:teuthology.orchestra.run.smithi046.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-06-23T01:00:50.189 INFO:teuthology.orchestra.run.smithi046.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c... 2023-06-23T01:00:50.733 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout f3276efde470d44ef0ff8e259464d1d34a5015beb9d7c1215520bcf3b48fd710 2023-06-23T01:00:50.733 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c... 2023-06-23T01:00:50.734 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Getting image source signatures 2023-06-23T01:00:50.734 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying blob sha256:9d60a6994110d194fea96cb2ff67b650891faff48e39f8a6b8ad8152d347e90d 2023-06-23T01:00:50.734 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying blob sha256:3dbfccf0662dd88e0c2efa9d59916ea5c98f7c5a05bbe18e86c2dc32f0494158 2023-06-23T01:00:50.734 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying config sha256:f3276efde470d44ef0ff8e259464d1d34a5015beb9d7c1215520bcf3b48fd710 2023-06-23T01:00:50.734 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Writing manifest to image destination 2023-06-23T01:00:50.734 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Storing signatures 2023-06-23T01:00:51.777 INFO:teuthology.orchestra.run.smithi046.stdout:ceph: stdout ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable) 2023-06-23T01:00:51.777 INFO:teuthology.orchestra.run.smithi046.stdout:Ceph version: ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable) 2023-06-23T01:00:51.777 INFO:teuthology.orchestra.run.smithi046.stdout:Extracting ceph user uid/gid from container image... 2023-06-23T01:00:52.725 INFO:teuthology.orchestra.run.smithi046.stdout:stat: stdout 167 167 2023-06-23T01:00:52.725 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial keys... 2023-06-23T01:00:53.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQDF7pRktU4fBhAAgi4x6W6X2ZzkngLQ3mSl1Q== 2023-06-23T01:00:54.511 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQDF7pRkWudvOxAA24ueASOXpjwqYn3jxqGljg== 2023-06-23T01:00:55.260 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQDG7pRkIsZOMhAASjgicrfeMaCJpNYud+efxA== 2023-06-23T01:00:55.261 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial monmap... 2023-06-23T01:00:55.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-06-23T01:00:55.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2023-06-23T01:00:55.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:00:55.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-06-23T01:00:55.910 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool for a [v2:172.21.15.46:3300,v1:172.21.15.46:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-06-23T01:00:55.911 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = octopus 2023-06-23T01:00:55.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: set fsid to 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:00:55.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-06-23T01:00:55.911 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:00:55.911 INFO:teuthology.orchestra.run.smithi046.stdout:Creating mon... 2023-06-23T01:00:56.755 INFO:teuthology.orchestra.run.smithi046.stdout:create mon.a on 2023-06-23T01:00:56.862 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-06-23T01:00:56.961 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-06-23T01:00:57.071 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-458642ee-1161-11ee-9b2b-001a4aab830c.target → /etc/systemd/system/ceph-458642ee-1161-11ee-9b2b-001a4aab830c.target. 2023-06-23T01:00:57.072 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-458642ee-1161-11ee-9b2b-001a4aab830c.target → /etc/systemd/system/ceph-458642ee-1161-11ee-9b2b-001a4aab830c.target. 2023-06-23T01:00:57.207 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.a 2023-06-23T01:00:57.208 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to reset failed state of unit ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.a.service: Unit ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.a.service not loaded. 2023-06-23T01:00:57.309 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-458642ee-1161-11ee-9b2b-001a4aab830c.target.wants/ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-458642ee-1161-11ee-9b2b-001a4aab830c@.service. 2023-06-23T01:00:57.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:57 smithi046 systemd[1]: Starting Ceph mon.a for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:00:57.946 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-06-23T01:00:57.947 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2023-06-23T01:00:57.954 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-06-23T01:00:57.955 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2023-06-23T01:00:57.955 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2023-06-23T01:00:57.955 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to enable service . firewalld.service is not available 2023-06-23T01:00:57.956 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mon to start... 2023-06-23T01:00:57.956 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mon... 2023-06-23T01:00:58.172 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:57 smithi046 bash[83957]: 05d807d358ff0b4016f22112e61fbcb22798f16f2c2f477f5bfe84c839a27897 2023-06-23T01:00:58.172 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:57 smithi046 ceph-mon[84021]: mkfs 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:00:58.172 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:57 smithi046 ceph-mon[84021]: mon.a is new leader, mons a in quorum (ranks 0) 2023-06-23T01:00:58.172 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:57 smithi046 systemd[1]: Started Ceph mon.a for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:00:58.900 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout cluster: 2023-06-23T01:00:58.900 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout id: 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:00:58.900 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-06-23T01:00:58.900 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:00:58.901 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout services: 2023-06-23T01:00:58.901 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.648284s) 2023-06-23T01:00:58.901 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-06-23T01:00:58.901 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-06-23T01:00:58.901 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:00:58.901 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout data: 2023-06-23T01:00:58.901 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-06-23T01:00:58.902 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-06-23T01:00:58.902 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-06-23T01:00:58.902 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout pgs: 2023-06-23T01:00:58.902 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:00:58.902 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.444+0000 7f34caf90700 1 Processor -- start 2023-06-23T01:00:58.902 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.444+0000 7f34caf90700 1 -- start start 2023-06-23T01:00:58.902 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.444+0000 7f34caf90700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4107190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:00:58.902 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.444+0000 7f34caf90700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c4102b80 con 0x7f34c4070db0 2023-06-23T01:00:58.903 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.444+0000 7f34c8d2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:00:58.903 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.444+0000 7f34c8d2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4107190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44920/0 (socket says 172.21.15.46:44920) 2023-06-23T01:00:58.903 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.444+0000 7f34c8d2c700 1 -- 172.21.15.46:0/786018969 learned_addr learned my addr 172.21.15.46:0/786018969 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:00:58.903 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.445+0000 7f34c8d2c700 1 -- 172.21.15.46:0/786018969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c41076d0 con 0x7f34c4070db0 2023-06-23T01:00:58.903 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.445+0000 7f34c8d2c700 1 --2- 172.21.15.46:0/786018969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4107190 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f34b80098e0 tx=0x7f34b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a82d0e31ab62fb4 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:00:58.903 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.445+0000 7f34c37fe700 1 -- 172.21.15.46:0/786018969 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34b8009e80 con 0x7f34c4070db0 2023-06-23T01:00:58.903 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.445+0000 7f34c37fe700 1 -- 172.21.15.46:0/786018969 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b800b860 con 0x7f34c4070db0 2023-06-23T01:00:58.903 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.445+0000 7f34c37fe700 1 -- 172.21.15.46:0/786018969 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34b800b9d0 con 0x7f34c4070db0 2023-06-23T01:00:58.904 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.446+0000 7f34caf90700 1 -- 172.21.15.46:0/786018969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 msgr2=0x7f34c4107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:00:58.904 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.446+0000 7f34caf90700 1 --2- 172.21.15.46:0/786018969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4107190 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f34b80098e0 tx=0x7f34b8009250 comp rx=0 tx=0).stop 2023-06-23T01:00:58.904 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.446+0000 7f34caf90700 1 -- 172.21.15.46:0/786018969 shutdown_connections 2023-06-23T01:00:58.904 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.446+0000 7f34caf90700 1 --2- 172.21.15.46:0/786018969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4107190 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:00:58.904 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.446+0000 7f34caf90700 1 -- 172.21.15.46:0/786018969 >> 172.21.15.46:0/786018969 conn(0x7f34c40fc800 msgr2=0x7f34c40fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:00:58.904 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.446+0000 7f34caf90700 1 -- 172.21.15.46:0/786018969 shutdown_connections 2023-06-23T01:00:58.904 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.446+0000 7f34caf90700 1 -- 172.21.15.46:0/786018969 wait complete. 2023-06-23T01:00:58.905 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.446+0000 7f34caf90700 1 Processor -- start 2023-06-23T01:00:58.905 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.446+0000 7f34caf90700 1 -- start start 2023-06-23T01:00:58.905 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34caf90700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4112880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:00:58.905 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34caf90700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c4112dc0 con 0x7f34c4070db0 2023-06-23T01:00:58.905 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34c8d2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4112880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:00:58.905 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34c8d2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4112880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44922/0 (socket says 172.21.15.46:44922) 2023-06-23T01:00:58.905 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34c8d2c700 1 -- 172.21.15.46:0/1169575910 learned_addr learned my addr 172.21.15.46:0/1169575910 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:00:58.905 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34c8d2c700 1 -- 172.21.15.46:0/1169575910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b80094f0 con 0x7f34c4070db0 2023-06-23T01:00:58.906 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34c8d2c700 1 --2- 172.21.15.46:0/1169575910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4112880 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009290 tx=0x7f34b8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:00:58.906 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34c17fa700 1 -- 172.21.15.46:0/1169575910 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34b801c460 con 0x7f34c4070db0 2023-06-23T01:00:58.906 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34c17fa700 1 -- 172.21.15.46:0/1169575910 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b801c5c0 con 0x7f34c4070db0 2023-06-23T01:00:58.906 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34caf90700 1 -- 172.21.15.46:0/1169575910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c4112fc0 con 0x7f34c4070db0 2023-06-23T01:00:58.906 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34c17fa700 1 -- 172.21.15.46:0/1169575910 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34b801c8b0 con 0x7f34c4070db0 2023-06-23T01:00:58.906 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.447+0000 7f34caf90700 1 -- 172.21.15.46:0/1169575910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c41133d0 con 0x7f34c4070db0 2023-06-23T01:00:58.906 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.448+0000 7f34c17fa700 1 -- 172.21.15.46:0/1169575910 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f34b801cc90 con 0x7f34c4070db0 2023-06-23T01:00:58.907 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.448+0000 7f34c17fa700 1 -- 172.21.15.46:0/1169575910 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f34b801ad70 con 0x7f34c4070db0 2023-06-23T01:00:58.907 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.448+0000 7f34caf90700 1 -- 172.21.15.46:0/1169575910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c404b970 con 0x7f34c4070db0 2023-06-23T01:00:58.907 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.451+0000 7f34c17fa700 1 -- 172.21.15.46:0/1169575910 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f34b801a450 con 0x7f34c4070db0 2023-06-23T01:00:58.907 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.523+0000 7f34caf90700 1 -- 172.21.15.46:0/1169575910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f34c410c170 con 0x7f34c4070db0 2023-06-23T01:00:58.907 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.523+0000 7f34c17fa700 1 -- 172.21.15.46:0/1169575910 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f34b8025440 con 0x7f34c4070db0 2023-06-23T01:00:58.907 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.524+0000 7f34caf90700 1 -- 172.21.15.46:0/1169575910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 msgr2=0x7f34c4112880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:00:58.908 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.524+0000 7f34caf90700 1 --2- 172.21.15.46:0/1169575910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4112880 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009290 tx=0x7f34b8016000 comp rx=0 tx=0).stop 2023-06-23T01:00:58.908 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.524+0000 7f34caf90700 1 -- 172.21.15.46:0/1169575910 shutdown_connections 2023-06-23T01:00:58.908 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.524+0000 7f34caf90700 1 --2- 172.21.15.46:0/1169575910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c4070db0 0x7f34c4112880 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:00:58.908 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.524+0000 7f34caf90700 1 -- 172.21.15.46:0/1169575910 >> 172.21.15.46:0/1169575910 conn(0x7f34c40fc800 msgr2=0x7f34c40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:00:58.908 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.524+0000 7f34caf90700 1 -- 172.21.15.46:0/1169575910 shutdown_connections 2023-06-23T01:00:58.908 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:58.524+0000 7f34caf90700 1 -- 172.21.15.46:0/1169575910 wait complete. 2023-06-23T01:00:58.908 INFO:teuthology.orchestra.run.smithi046.stdout:mon is available 2023-06-23T01:00:58.909 INFO:teuthology.orchestra.run.smithi046.stdout:Assimilating anything we can from ceph.conf... 2023-06-23T01:00:59.154 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:58 smithi046 ceph-mon[84021]: mon.a is new leader, mons a in quorum (ranks 0) 2023-06-23T01:00:59.154 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:58 smithi046 ceph-mon[84021]: monmap e1: 1 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0]} removed_ranks: {} 2023-06-23T01:00:59.154 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:58 smithi046 ceph-mon[84021]: fsmap 2023-06-23T01:00:59.154 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:58 smithi046 ceph-mon[84021]: osdmap e1: 0 total, 0 up, 0 in 2023-06-23T01:00:59.154 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:58 smithi046 ceph-mon[84021]: mgrmap e1: no daemons active 2023-06-23T01:00:59.155 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:58 smithi046 ceph-mon[84021]: from='client.? 172.21.15.46:0/1169575910' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-06-23T01:00:59.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:00:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [global] 2023-06-23T01:00:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout fsid = 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:00:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2023-06-23T01:00:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-06-23T01:00:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-06-23T01:00:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-06-23T01:00:59.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-06-23T01:00:59.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:00:59.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [mgr] 2023-06-23T01:00:59.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-06-23T01:00:59.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:00:59.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [osd] 2023-06-23T01:00:59.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2023-06-23T01:00:59.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-06-23T01:00:59.944 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-06-23T01:00:59.944 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-06-23T01:00:59.944 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-06-23T01:00:59.944 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-06-23T01:00:59.944 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.426+0000 7f3deff2d700 1 Processor -- start 2023-06-23T01:00:59.945 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.426+0000 7f3deff2d700 1 -- start start 2023-06-23T01:00:59.945 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.426+0000 7f3deff2d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de8101490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:00:59.945 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.426+0000 7f3deff2d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de81019d0 con 0x7f3de8101080 2023-06-23T01:00:59.945 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.427+0000 7f3dedcc9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de8101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:00:59.945 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.427+0000 7f3dedcc9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de8101490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44928/0 (socket says 172.21.15.46:44928) 2023-06-23T01:00:59.945 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.427+0000 7f3dedcc9700 1 -- 172.21.15.46:0/3524029919 learned_addr learned my addr 172.21.15.46:0/3524029919 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:00:59.945 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.427+0000 7f3dedcc9700 1 -- 172.21.15.46:0/3524029919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de8101b10 con 0x7f3de8101080 2023-06-23T01:00:59.946 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.427+0000 7f3dedcc9700 1 --2- 172.21.15.46:0/3524029919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de8101490 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80098e0 tx=0x7f3dd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491efaa0fd362724 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:00:59.946 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.427+0000 7f3deccc7700 1 -- 172.21.15.46:0/3524029919 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd8009e80 con 0x7f3de8101080 2023-06-23T01:00:59.946 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.427+0000 7f3deccc7700 1 -- 172.21.15.46:0/3524029919 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd800b860 con 0x7f3de8101080 2023-06-23T01:00:59.946 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.427+0000 7f3deccc7700 1 -- 172.21.15.46:0/3524029919 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd800b9d0 con 0x7f3de8101080 2023-06-23T01:00:59.946 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.428+0000 7f3deff2d700 1 -- 172.21.15.46:0/3524029919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 msgr2=0x7f3de8101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:00:59.946 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.428+0000 7f3deff2d700 1 --2- 172.21.15.46:0/3524029919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de8101490 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3dd80098e0 tx=0x7f3dd8009250 comp rx=0 tx=0).stop 2023-06-23T01:00:59.946 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.428+0000 7f3deff2d700 1 -- 172.21.15.46:0/3524029919 shutdown_connections 2023-06-23T01:00:59.946 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.428+0000 7f3deff2d700 1 --2- 172.21.15.46:0/3524029919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de8101490 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:00:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.428+0000 7f3deff2d700 1 -- 172.21.15.46:0/3524029919 >> 172.21.15.46:0/3524029919 conn(0x7f3de80fc800 msgr2=0x7f3de80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:00:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.428+0000 7f3deff2d700 1 -- 172.21.15.46:0/3524029919 shutdown_connections 2023-06-23T01:00:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.428+0000 7f3deff2d700 1 -- 172.21.15.46:0/3524029919 wait complete. 2023-06-23T01:00:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.428+0000 7f3deff2d700 1 Processor -- start 2023-06-23T01:00:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3deff2d700 1 -- start start 2023-06-23T01:00:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3deff2d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de810f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:00:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3deff2d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de810fad0 con 0x7f3de8101080 2023-06-23T01:00:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3dedcc9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de810f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:00:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3dedcc9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de810f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44938/0 (socket says 172.21.15.46:44938) 2023-06-23T01:00:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3dedcc9700 1 -- 172.21.15.46:0/223587150 learned_addr learned my addr 172.21.15.46:0/223587150 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:00:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3dedcc9700 1 -- 172.21.15.46:0/223587150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd80094f0 con 0x7f3de8101080 2023-06-23T01:00:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3dedcc9700 1 --2- 172.21.15.46:0/223587150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de810f590 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8003080 tx=0x7f3dd8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:00:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3dde7fc700 1 -- 172.21.15.46:0/223587150 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd801c460 con 0x7f3de8101080 2023-06-23T01:00:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3dde7fc700 1 -- 172.21.15.46:0/223587150 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd801c5c0 con 0x7f3de8101080 2023-06-23T01:00:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3deff2d700 1 -- 172.21.15.46:0/223587150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de810fcd0 con 0x7f3de8101080 2023-06-23T01:00:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3dde7fc700 1 -- 172.21.15.46:0/223587150 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3dd801c890 con 0x7f3de8101080 2023-06-23T01:00:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.429+0000 7f3deff2d700 1 -- 172.21.15.46:0/223587150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de81100e0 con 0x7f3de8101080 2023-06-23T01:00:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.430+0000 7f3dde7fc700 1 -- 172.21.15.46:0/223587150 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3dd801a450 con 0x7f3de8101080 2023-06-23T01:00:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.430+0000 7f3dde7fc700 1 -- 172.21.15.46:0/223587150 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3dd802e790 con 0x7f3de8101080 2023-06-23T01:00:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.430+0000 7f3deff2d700 1 -- 172.21.15.46:0/223587150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de804b970 con 0x7f3de8101080 2023-06-23T01:00:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.433+0000 7f3dde7fc700 1 -- 172.21.15.46:0/223587150 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f3dd801cd80 con 0x7f3de8101080 2023-06-23T01:00:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.502+0000 7f3deff2d700 1 -- 172.21.15.46:0/223587150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3de8110b90 con 0x7f3de8101080 2023-06-23T01:00:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.505+0000 7f3dde7fc700 1 -- 172.21.15.46:0/223587150 <== mon.0 v2:172.21.15.46:3300/0 7 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f3dd8006d70 con 0x7f3de8101080 2023-06-23T01:00:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.505+0000 7f3dde7fc700 1 -- 172.21.15.46:0/223587150 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+454 (secure 0 0 0) 0x7f3dd801b520 con 0x7f3de8101080 2023-06-23T01:00:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.506+0000 7f3deff2d700 1 -- 172.21.15.46:0/223587150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 msgr2=0x7f3de810f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:00:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.506+0000 7f3deff2d700 1 --2- 172.21.15.46:0/223587150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de810f590 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8003080 tx=0x7f3dd8016000 comp rx=0 tx=0).stop 2023-06-23T01:00:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.506+0000 7f3deff2d700 1 -- 172.21.15.46:0/223587150 shutdown_connections 2023-06-23T01:00:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.506+0000 7f3deff2d700 1 --2- 172.21.15.46:0/223587150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3de8101080 0x7f3de810f590 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:00:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.506+0000 7f3deff2d700 1 -- 172.21.15.46:0/223587150 >> 172.21.15.46:0/223587150 conn(0x7f3de80fc800 msgr2=0x7f3de8103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:00:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.506+0000 7f3deff2d700 1 -- 172.21.15.46:0/223587150 shutdown_connections 2023-06-23T01:00:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:00:59.506+0000 7f3deff2d700 1 -- 172.21.15.46:0/223587150 wait complete. 2023-06-23T01:00:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:Generating new minimal ceph.conf... 2023-06-23T01:01:00.170 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:59 smithi046 ceph-mon[84021]: from='client.? 172.21.15.46:0/223587150' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-06-23T01:01:00.170 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:00:59 smithi046 ceph-mon[84021]: from='client.? 172.21.15.46:0/223587150' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-06-23T01:01:01.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:00 smithi046 ceph-mon[84021]: from='client.? 172.21.15.46:0/1527060751' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:01:02.077 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.656+0000 7f4e331ee700 1 Processor -- start 2023-06-23T01:01:02.080 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.657+0000 7f4e331ee700 1 -- start start 2023-06-23T01:01:02.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.657+0000 7f4e331ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c101490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:02.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.657+0000 7f4e331ee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c1019d0 con 0x7f4e2c101080 2023-06-23T01:01:02.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.657+0000 7f4e30f8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:02.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.657+0000 7f4e30f8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c101490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44940/0 (socket says 172.21.15.46:44940) 2023-06-23T01:01:02.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.657+0000 7f4e30f8a700 1 -- 172.21.15.46:0/307358966 learned_addr learned my addr 172.21.15.46:0/307358966 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:02.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.657+0000 7f4e30f8a700 1 -- 172.21.15.46:0/307358966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c101b10 con 0x7f4e2c101080 2023-06-23T01:01:02.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.657+0000 7f4e30f8a700 1 --2- 172.21.15.46:0/307358966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c101490 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000bb80 tx=0x7f4e2000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d604533e93dc7795 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:02.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.657+0000 7f4e2b7fe700 1 -- 172.21.15.46:0/307358966 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e2000dba0 con 0x7f4e2c101080 2023-06-23T01:01:02.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.658+0000 7f4e2b7fe700 1 -- 172.21.15.46:0/307358966 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4e2000dd00 con 0x7f4e2c101080 2023-06-23T01:01:02.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.658+0000 7f4e2b7fe700 1 -- 172.21.15.46:0/307358966 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e20018670 con 0x7f4e2c101080 2023-06-23T01:01:02.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.658+0000 7f4e331ee700 1 -- 172.21.15.46:0/307358966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 msgr2=0x7f4e2c101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:02.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.658+0000 7f4e331ee700 1 --2- 172.21.15.46:0/307358966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c101490 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000bb80 tx=0x7f4e2000b4f0 comp rx=0 tx=0).stop 2023-06-23T01:01:02.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.658+0000 7f4e331ee700 1 -- 172.21.15.46:0/307358966 shutdown_connections 2023-06-23T01:01:02.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.658+0000 7f4e331ee700 1 --2- 172.21.15.46:0/307358966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c101490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:02.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.658+0000 7f4e331ee700 1 -- 172.21.15.46:0/307358966 >> 172.21.15.46:0/307358966 conn(0x7f4e2c0fc800 msgr2=0x7f4e2c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:02.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.658+0000 7f4e331ee700 1 -- 172.21.15.46:0/307358966 shutdown_connections 2023-06-23T01:01:02.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.658+0000 7f4e331ee700 1 -- 172.21.15.46:0/307358966 wait complete. 2023-06-23T01:01:02.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.659+0000 7f4e331ee700 1 Processor -- start 2023-06-23T01:01:02.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.659+0000 7f4e331ee700 1 -- start start 2023-06-23T01:01:02.085 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.659+0000 7f4e331ee700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c10eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:02.085 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.659+0000 7f4e331ee700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c10f0a0 con 0x7f4e2c101080 2023-06-23T01:01:02.085 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.659+0000 7f4e30f8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c10eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:02.085 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.659+0000 7f4e30f8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c10eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44946/0 (socket says 172.21.15.46:44946) 2023-06-23T01:01:02.086 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.659+0000 7f4e30f8a700 1 -- 172.21.15.46:0/1527060751 learned_addr learned my addr 172.21.15.46:0/1527060751 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:02.086 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.659+0000 7f4e30f8a700 1 -- 172.21.15.46:0/1527060751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2000b790 con 0x7f4e2c101080 2023-06-23T01:01:02.086 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.659+0000 7f4e30f8a700 1 --2- 172.21.15.46:0/1527060751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c10eb60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4e20003080 tx=0x7f4e2000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:02.086 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.660+0000 7f4e297fa700 1 -- 172.21.15.46:0/1527060751 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e20010750 con 0x7f4e2c101080 2023-06-23T01:01:02.087 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.660+0000 7f4e297fa700 1 -- 172.21.15.46:0/1527060751 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4e2001d050 con 0x7f4e2c101080 2023-06-23T01:01:02.087 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.660+0000 7f4e331ee700 1 -- 172.21.15.46:0/1527060751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2c10f2a0 con 0x7f4e2c101080 2023-06-23T01:01:02.087 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.660+0000 7f4e297fa700 1 -- 172.21.15.46:0/1527060751 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e200223d0 con 0x7f4e2c101080 2023-06-23T01:01:02.087 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.660+0000 7f4e331ee700 1 -- 172.21.15.46:0/1527060751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c10f6b0 con 0x7f4e2c101080 2023-06-23T01:01:02.088 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.660+0000 7f4e297fa700 1 -- 172.21.15.46:0/1527060751 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4e20022530 con 0x7f4e2c101080 2023-06-23T01:01:02.088 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.660+0000 7f4e297fa700 1 -- 172.21.15.46:0/1527060751 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4e20014070 con 0x7f4e2c101080 2023-06-23T01:01:02.088 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.661+0000 7f4e331ee700 1 -- 172.21.15.46:0/1527060751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2c04b970 con 0x7f4e2c101080 2023-06-23T01:01:02.092 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.663+0000 7f4e297fa700 1 -- 172.21.15.46:0/1527060751 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f4e20010b20 con 0x7f4e2c101080 2023-06-23T01:01:02.092 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.734+0000 7f4e331ee700 1 -- 172.21.15.46:0/1527060751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f4e2c107f80 con 0x7f4e2c101080 2023-06-23T01:01:02.092 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.734+0000 7f4e297fa700 1 -- 172.21.15.46:0/1527060751 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f4e200108b0 con 0x7f4e2c101080 2023-06-23T01:01:02.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.735+0000 7f4e331ee700 1 -- 172.21.15.46:0/1527060751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 msgr2=0x7f4e2c10eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:02.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.735+0000 7f4e331ee700 1 --2- 172.21.15.46:0/1527060751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c10eb60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4e20003080 tx=0x7f4e2000b4f0 comp rx=0 tx=0).stop 2023-06-23T01:01:02.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.735+0000 7f4e331ee700 1 -- 172.21.15.46:0/1527060751 shutdown_connections 2023-06-23T01:01:02.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.735+0000 7f4e331ee700 1 --2- 172.21.15.46:0/1527060751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e2c101080 0x7f4e2c10eb60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:02.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.735+0000 7f4e331ee700 1 -- 172.21.15.46:0/1527060751 >> 172.21.15.46:0/1527060751 conn(0x7f4e2c0fc800 msgr2=0x7f4e2c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:02.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.735+0000 7f4e331ee700 1 -- 172.21.15.46:0/1527060751 shutdown_connections 2023-06-23T01:01:02.093 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:00.735+0000 7f4e331ee700 1 -- 172.21.15.46:0/1527060751 wait complete. 2023-06-23T01:01:02.094 INFO:teuthology.orchestra.run.smithi046.stdout:Restarting the monitor... 2023-06-23T01:01:02.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:02 smithi046 systemd[1]: Stopping Ceph mon.a for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:01:03.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:03 smithi046 conmon[83997]: 2023-06-23T01:01:03.052+0000 7f345017e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:01:03.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:03 smithi046 conmon[83997]: 2023-06-23T01:01:03.052+0000 7f345017e700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-06-23T01:01:04.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:03 smithi046 bash[84563]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mon-a 2023-06-23T01:01:05.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:05 smithi046 systemd[1]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.a.service: Succeeded. 2023-06-23T01:01:05.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:05 smithi046 systemd[1]: Stopped Ceph mon.a for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:01:05.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:05 smithi046 systemd[1]: Starting Ceph mon.a for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:01:07.616 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: set uid:gid to 167:167 (ceph:ceph) 2023-06-23T01:01:07.617 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable), process ceph-mon, pid 7 2023-06-23T01:01:07.617 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: pidfile_write: ignore empty --pid-file 2023-06-23T01:01:07.618 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: load: jerasure load: lrc 2023-06-23T01:01:07.618 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: RocksDB version: 6.15.5 2023-06-23T01:01:07.618 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-06-23T01:01:07.618 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Compile date Jun 21 2023 2023-06-23T01:01:07.619 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: DB SUMMARY 2023-06-23T01:01:07.619 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: DB Session ID: TKKT910VUBG29XS4FAVO 2023-06-23T01:01:07.619 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: CURRENT file: CURRENT 2023-06-23T01:01:07.619 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: IDENTITY file: IDENTITY 2023-06-23T01:01:07.620 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2023-06-23T01:01:07.620 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-06-23T01:01:07.620 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 97271 ; 2023-06-23T01:01:07.621 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.error_if_exists: 0 2023-06-23T01:01:07.621 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.create_if_missing: 0 2023-06-23T01:01:07.621 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.paranoid_checks: 1 2023-06-23T01:01:07.621 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-06-23T01:01:07.622 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.env: 0x558eeb228a80 2023-06-23T01:01:07.622 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.fs: Posix File System 2023-06-23T01:01:07.622 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.info_log: 0x558eec0bfda0 2023-06-23T01:01:07.622 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_file_opening_threads: 16 2023-06-23T01:01:07.623 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.statistics: (nil) 2023-06-23T01:01:07.623 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.use_fsync: 0 2023-06-23T01:01:07.623 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_log_file_size: 0 2023-06-23T01:01:07.623 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-06-23T01:01:07.623 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.log_file_time_to_roll: 0 2023-06-23T01:01:07.624 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.keep_log_file_num: 1000 2023-06-23T01:01:07.624 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.recycle_log_file_num: 0 2023-06-23T01:01:07.624 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.allow_fallocate: 1 2023-06-23T01:01:07.624 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.allow_mmap_reads: 0 2023-06-23T01:01:07.625 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.allow_mmap_writes: 0 2023-06-23T01:01:07.625 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.use_direct_reads: 0 2023-06-23T01:01:07.625 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-06-23T01:01:07.625 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.create_missing_column_families: 0 2023-06-23T01:01:07.626 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.db_log_dir: 2023-06-23T01:01:07.626 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2023-06-23T01:01:07.626 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.table_cache_numshardbits: 6 2023-06-23T01:01:07.626 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.WAL_ttl_seconds: 0 2023-06-23T01:01:07.627 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.WAL_size_limit_MB: 0 2023-06-23T01:01:07.627 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-06-23T01:01:07.627 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-06-23T01:01:07.627 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.is_fd_close_on_exec: 1 2023-06-23T01:01:07.627 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.advise_random_on_open: 1 2023-06-23T01:01:07.628 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.db_write_buffer_size: 0 2023-06-23T01:01:07.628 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.write_buffer_manager: 0x558eec184930 2023-06-23T01:01:07.630 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-06-23T01:01:07.630 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-06-23T01:01:07.630 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-06-23T01:01:07.630 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.use_adaptive_mutex: 0 2023-06-23T01:01:07.631 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.rate_limiter: (nil) 2023-06-23T01:01:07.631 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-06-23T01:01:07.631 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.wal_recovery_mode: 2 2023-06-23T01:01:07.632 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.enable_thread_tracking: 0 2023-06-23T01:01:07.632 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.enable_pipelined_write: 0 2023-06-23T01:01:07.632 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.unordered_write: 0 2023-06-23T01:01:07.632 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-06-23T01:01:07.633 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-06-23T01:01:07.633 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-06-23T01:01:07.633 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-06-23T01:01:07.634 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.row_cache: None 2023-06-23T01:01:07.634 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.wal_filter: None 2023-06-23T01:01:07.634 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-06-23T01:01:07.635 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.allow_ingest_behind: 0 2023-06-23T01:01:07.635 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.preserve_deletes: 0 2023-06-23T01:01:07.635 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.two_write_queues: 0 2023-06-23T01:01:07.635 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.manual_wal_flush: 0 2023-06-23T01:01:07.636 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.atomic_flush: 0 2023-06-23T01:01:07.636 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-06-23T01:01:07.636 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.persist_stats_to_disk: 0 2023-06-23T01:01:07.637 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.write_dbid_to_manifest: 0 2023-06-23T01:01:07.637 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.log_readahead_size: 0 2023-06-23T01:01:07.637 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-06-23T01:01:07.637 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.best_efforts_recovery: 0 2023-06-23T01:01:07.638 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-06-23T01:01:07.638 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-06-23T01:01:07.638 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.allow_data_in_errors: 0 2023-06-23T01:01:07.639 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.db_host_id: __hostname__ 2023-06-23T01:01:07.639 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_background_jobs: 2 2023-06-23T01:01:07.641 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_background_compactions: -1 2023-06-23T01:01:07.641 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_subcompactions: 1 2023-06-23T01:01:07.642 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-06-23T01:01:07.642 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-06-23T01:01:07.642 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.delayed_write_rate : 16777216 2023-06-23T01:01:07.642 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_total_wal_size: 0 2023-06-23T01:01:07.642 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-06-23T01:01:07.642 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.stats_dump_period_sec: 600 2023-06-23T01:01:07.642 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.stats_persist_period_sec: 600 2023-06-23T01:01:07.643 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-06-23T01:01:07.643 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_open_files: -1 2023-06-23T01:01:07.643 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.bytes_per_sync: 0 2023-06-23T01:01:07.643 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.wal_bytes_per_sync: 0 2023-06-23T01:01:07.643 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.strict_bytes_per_sync: 0 2023-06-23T01:01:07.643 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_readahead_size: 0 2023-06-23T01:01:07.643 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_background_flushes: -1 2023-06-23T01:01:07.644 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Compression algorithms supported: 2023-06-23T01:01:07.644 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-06-23T01:01:07.644 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: kZSTD supported: 0 2023-06-23T01:01:07.644 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: kXpressCompression supported: 0 2023-06-23T01:01:07.644 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: kLZ4HCCompression supported: 1 2023-06-23T01:01:07.644 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: kZlibCompression supported: 1 2023-06-23T01:01:07.644 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: kSnappyCompression supported: 1 2023-06-23T01:01:07.645 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: kLZ4Compression supported: 1 2023-06-23T01:01:07.645 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: kBZip2Compression supported: 0 2023-06-23T01:01:07.645 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Fast CRC32 supported: Supported on x86 2023-06-23T01:01:07.645 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2023-06-23T01:01:07.645 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-06-23T01:01:07.645 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-06-23T01:01:07.645 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.merge_operator: 2023-06-23T01:01:07.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_filter: None 2023-06-23T01:01:07.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_filter_factory: None 2023-06-23T01:01:07.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.sst_partitioner_factory: None 2023-06-23T01:01:07.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.memtable_factory: SkipListFactory 2023-06-23T01:01:07.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.table_factory: BlockBasedTable 2023-06-23T01:01:07.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558eec08ed30) 2023-06-23T01:01:07.647 INFO:journalctl@ceph.mon.a.smithi046.stdout: cache_index_and_filter_blocks: 1 2023-06-23T01:01:07.647 INFO:journalctl@ceph.mon.a.smithi046.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-06-23T01:01:07.647 INFO:journalctl@ceph.mon.a.smithi046.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-06-23T01:01:07.647 INFO:journalctl@ceph.mon.a.smithi046.stdout: pin_top_level_index_and_filter: 1 2023-06-23T01:01:07.648 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_type: 0 2023-06-23T01:01:07.648 INFO:journalctl@ceph.mon.a.smithi046.stdout: data_block_index_type: 0 2023-06-23T01:01:07.648 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_shortening: 1 2023-06-23T01:01:07.648 INFO:journalctl@ceph.mon.a.smithi046.stdout: data_block_hash_table_util_ratio: 0.750000 2023-06-23T01:01:07.648 INFO:journalctl@ceph.mon.a.smithi046.stdout: hash_index_allow_collision: 1 2023-06-23T01:01:07.648 INFO:journalctl@ceph.mon.a.smithi046.stdout: checksum: 1 2023-06-23T01:01:07.649 INFO:journalctl@ceph.mon.a.smithi046.stdout: no_block_cache: 0 2023-06-23T01:01:07.649 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache: 0x558eec1471f0 2023-06-23T01:01:07.649 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_name: BinnedLRUCache 2023-06-23T01:01:07.649 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_options: 2023-06-23T01:01:07.650 INFO:journalctl@ceph.mon.a.smithi046.stdout: capacity : 536870912 2023-06-23T01:01:07.650 INFO:journalctl@ceph.mon.a.smithi046.stdout: num_shard_bits : 4 2023-06-23T01:01:07.650 INFO:journalctl@ceph.mon.a.smithi046.stdout: strict_capacity_limit : 0 2023-06-23T01:01:07.650 INFO:journalctl@ceph.mon.a.smithi046.stdout: high_pri_pool_ratio: 0.000 2023-06-23T01:01:07.651 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_compressed: (nil) 2023-06-23T01:01:07.651 INFO:journalctl@ceph.mon.a.smithi046.stdout: persistent_cache: (nil) 2023-06-23T01:01:07.651 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_size: 4096 2023-06-23T01:01:07.651 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_size_deviation: 10 2023-06-23T01:01:07.651 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_restart_interval: 16 2023-06-23T01:01:07.651 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_block_restart_interval: 1 2023-06-23T01:01:07.651 INFO:journalctl@ceph.mon.a.smithi046.stdout: metadata_block_size: 4096 2023-06-23T01:01:07.652 INFO:journalctl@ceph.mon.a.smithi046.stdout: partition_filters: 0 2023-06-23T01:01:07.652 INFO:journalctl@ceph.mon.a.smithi046.stdout: use_delta_encoding: 1 2023-06-23T01:01:07.652 INFO:journalctl@ceph.mon.a.smithi046.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2023-06-23T01:01:07.652 INFO:journalctl@ceph.mon.a.smithi046.stdout: whole_key_filtering: 1 2023-06-23T01:01:07.652 INFO:journalctl@ceph.mon.a.smithi046.stdout: verify_compression: 0 2023-06-23T01:01:07.652 INFO:journalctl@ceph.mon.a.smithi046.stdout: read_amp_bytes_per_bit: 0 2023-06-23T01:01:07.652 INFO:journalctl@ceph.mon.a.smithi046.stdout: format_version: 4 2023-06-23T01:01:07.653 INFO:journalctl@ceph.mon.a.smithi046.stdout: enable_index_compression: 1 2023-06-23T01:01:07.653 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_align: 0 2023-06-23T01:01:07.653 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.write_buffer_size: 33554432 2023-06-23T01:01:07.653 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_write_buffer_number: 2 2023-06-23T01:01:07.653 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compression: NoCompression 2023-06-23T01:01:07.653 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.bottommost_compression: Disabled 2023-06-23T01:01:07.653 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.prefix_extractor: nullptr 2023-06-23T01:01:07.653 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-06-23T01:01:07.654 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.num_levels: 7 2023-06-23T01:01:07.654 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-06-23T01:01:07.654 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-06-23T01:01:07.654 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-06-23T01:01:07.654 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-06-23T01:01:07.655 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-06-23T01:01:07.655 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-06-23T01:01:07.655 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-06-23T01:01:07.655 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-06-23T01:01:07.655 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-06-23T01:01:07.655 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-06-23T01:01:07.655 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compression_opts.window_bits: -14 2023-06-23T01:01:07.655 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compression_opts.level: 32767 2023-06-23T01:01:07.656 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compression_opts.strategy: 0 2023-06-23T01:01:07.656 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-06-23T01:01:07.656 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-06-23T01:01:07.656 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-06-23T01:01:07.657 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compression_opts.enabled: false 2023-06-23T01:01:07.657 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-06-23T01:01:07.657 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-06-23T01:01:07.657 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-06-23T01:01:07.657 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.target_file_size_base: 67108864 2023-06-23T01:01:07.658 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.target_file_size_multiplier: 1 2023-06-23T01:01:07.658 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-06-23T01:01:07.658 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-06-23T01:01:07.658 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-06-23T01:01:07.659 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-06-23T01:01:07.659 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-06-23T01:01:07.659 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-06-23T01:01:07.659 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-06-23T01:01:07.659 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-06-23T01:01:07.660 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-06-23T01:01:07.660 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-06-23T01:01:07.660 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-06-23T01:01:07.660 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-06-23T01:01:07.661 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.arena_block_size: 4194304 2023-06-23T01:01:07.661 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-06-23T01:01:07.661 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-06-23T01:01:07.661 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-06-23T01:01:07.661 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.disable_auto_compactions: 0 2023-06-23T01:01:07.661 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-06-23T01:01:07.661 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-06-23T01:01:07.662 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-06-23T01:01:07.662 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-06-23T01:01:07.662 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-06-23T01:01:07.662 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-06-23T01:01:07.662 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-06-23T01:01:07.662 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-06-23T01:01:07.662 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-06-23T01:01:07.663 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-06-23T01:01:07.663 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.table_properties_collectors: 2023-06-23T01:01:07.663 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.inplace_update_support: 0 2023-06-23T01:01:07.663 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.inplace_update_num_locks: 10000 2023-06-23T01:01:07.663 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-06-23T01:01:07.663 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-06-23T01:01:07.663 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.memtable_huge_page_size: 0 2023-06-23T01:01:07.663 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.bloom_locality: 0 2023-06-23T01:01:07.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.max_successive_merges: 0 2023-06-23T01:01:07.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.optimize_filters_for_hits: 0 2023-06-23T01:01:07.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.paranoid_file_checks: 0 2023-06-23T01:01:07.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.force_consistency_checks: 1 2023-06-23T01:01:07.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.report_bg_io_stats: 0 2023-06-23T01:01:07.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.ttl: 2592000 2023-06-23T01:01:07.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.periodic_compaction_seconds: 0 2023-06-23T01:01:07.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.enable_blob_files: false 2023-06-23T01:01:07.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.min_blob_size: 0 2023-06-23T01:01:07.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.blob_file_size: 268435456 2023-06-23T01:01:07.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.blob_compression_type: NoCompression 2023-06-23T01:01:07.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.enable_blob_garbage_collection: false 2023-06-23T01:01:07.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-06-23T01:01:07.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-06-23T01:01:07.666 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2023-06-23T01:01:07.666 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2023-06-23T01:01:07.666 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687482067356479, "job": 1, "event": "recovery_started", "wal_files": [10]} 2023-06-23T01:01:07.666 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2023-06-23T01:01:07.667 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-06-23T01:01:07.667 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687482067357343, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92335, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90430, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13271, "raw_average_key_size": 49, "raw_value_size": 83187, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 268, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1687482067, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "274f8e04-1162-4ac8-a558-2bb7bf731388", "db_session_id": "TKKT910VUBG29XS4FAVO"}} 2023-06-23T01:01:07.667 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2023-06-23T01:01:07.668 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687482067357778, "job": 1, "event": "recovery_finished"} 2023-06-23T01:01:07.668 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-06-23T01:01:07.668 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x558eec0db180 2023-06-23T01:01:07.669 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: DB pointer 0x558eec0f6000 2023-06-23T01:01:07.669 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-06-23T01:01:07.669 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: rocksdb: [db/db_impl/db_impl.cc:903] 2023-06-23T01:01:07.669 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** DB Stats ** 2023-06-23T01:01:07.669 INFO:journalctl@ceph.mon.a.smithi046.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-23T01:01:07.669 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-06-23T01:01:07.670 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-06-23T01:01:07.670 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-06-23T01:01:07.670 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-06-23T01:01:07.670 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-06-23T01:01:07.670 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-06-23T01:01:07.670 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2023-06-23T01:01:07.670 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** Compaction Stats [default] ** 2023-06-23T01:01:07.670 INFO:journalctl@ceph.mon.a.smithi046.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-23T01:01:07.671 INFO:journalctl@ceph.mon.a.smithi046.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-23T01:01:07.671 INFO:journalctl@ceph.mon.a.smithi046.stdout: L0 2/0 91.93 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.0 0.00 0.00 1 0.001 0 0 2023-06-23T01:01:07.671 INFO:journalctl@ceph.mon.a.smithi046.stdout: Sum 2/0 91.93 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.0 0.00 0.00 1 0.001 0 0 2023-06-23T01:01:07.671 INFO:journalctl@ceph.mon.a.smithi046.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.0 0.00 0.00 1 0.001 0 0 2023-06-23T01:01:07.671 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2023-06-23T01:01:07.671 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** Compaction Stats [default] ** 2023-06-23T01:01:07.671 INFO:journalctl@ceph.mon.a.smithi046.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-23T01:01:07.672 INFO:journalctl@ceph.mon.a.smithi046.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-23T01:01:07.672 INFO:journalctl@ceph.mon.a.smithi046.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.0 0.00 0.00 1 0.001 0 0 2023-06-23T01:01:07.672 INFO:journalctl@ceph.mon.a.smithi046.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-23T01:01:07.672 INFO:journalctl@ceph.mon.a.smithi046.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-06-23T01:01:07.672 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-06-23T01:01:07.672 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-06-23T01:01:07.672 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-06-23T01:01:07.672 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(Keys): cumulative 0, interval 0 2023-06-23T01:01:07.673 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative compaction: 0.00 GB write, 24.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-23T01:01:07.673 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval compaction: 0.00 GB write, 24.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-23T01:01:07.673 INFO:journalctl@ceph.mon.a.smithi046.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-06-23T01:01:07.673 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2023-06-23T01:01:07.673 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** File Read Latency Histogram By Level [default] ** 2023-06-23T01:01:07.673 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2023-06-23T01:01:07.673 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** Compaction Stats [default] ** 2023-06-23T01:01:07.674 INFO:journalctl@ceph.mon.a.smithi046.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-23T01:01:07.674 INFO:journalctl@ceph.mon.a.smithi046.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-23T01:01:07.674 INFO:journalctl@ceph.mon.a.smithi046.stdout: L0 2/0 91.93 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.0 0.00 0.00 1 0.001 0 0 2023-06-23T01:01:07.674 INFO:journalctl@ceph.mon.a.smithi046.stdout: Sum 2/0 91.93 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.0 0.00 0.00 1 0.001 0 0 2023-06-23T01:01:07.674 INFO:journalctl@ceph.mon.a.smithi046.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-06-23T01:01:07.674 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2023-06-23T01:01:07.674 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** Compaction Stats [default] ** 2023-06-23T01:01:07.674 INFO:journalctl@ceph.mon.a.smithi046.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-23T01:01:07.675 INFO:journalctl@ceph.mon.a.smithi046.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-23T01:01:07.675 INFO:journalctl@ceph.mon.a.smithi046.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.0 0.00 0.00 1 0.001 0 0 2023-06-23T01:01:07.675 INFO:journalctl@ceph.mon.a.smithi046.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-23T01:01:07.675 INFO:journalctl@ceph.mon.a.smithi046.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-06-23T01:01:07.675 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-06-23T01:01:07.675 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-06-23T01:01:07.675 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-06-23T01:01:07.676 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(Keys): cumulative 0, interval 0 2023-06-23T01:01:07.676 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative compaction: 0.00 GB write, 23.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-23T01:01:07.676 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-23T01:01:07.676 INFO:journalctl@ceph.mon.a.smithi046.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-06-23T01:01:07.676 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2023-06-23T01:01:07.676 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** File Read Latency Histogram By Level [default] ** 2023-06-23T01:01:07.676 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: starting mon.a rank 0 at public addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] at bind addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:01:07.676 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: mon.a@-1(???) e1 preinit fsid 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:01:07.677 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: mon.a@-1(???).mds e1 new map 2023-06-23T01:01:07.677 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: mon.a@-1(???).mds e1 print_map 2023-06-23T01:01:07.677 INFO:journalctl@ceph.mon.a.smithi046.stdout: e1 2023-06-23T01:01:07.677 INFO:journalctl@ceph.mon.a.smithi046.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-06-23T01:01:07.677 INFO:journalctl@ceph.mon.a.smithi046.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-06-23T01:01:07.677 INFO:journalctl@ceph.mon.a.smithi046.stdout: legacy client fscid: -1 2023-06-23T01:01:07.677 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2023-06-23T01:01:07.678 INFO:journalctl@ceph.mon.a.smithi046.stdout: No filesystems configured 2023-06-23T01:01:07.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-06-23T01:01:07.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-06-23T01:01:07.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-06-23T01:01:07.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-06-23T01:01:07.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-06-23T01:01:07.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: mon.a is new leader, mons a in quorum (ranks 0) 2023-06-23T01:01:07.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: monmap e1: 1 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0]} removed_ranks: {} 2023-06-23T01:01:07.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: fsmap 2023-06-23T01:01:07.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: osdmap e1: 0 total, 0 up, 0 in 2023-06-23T01:01:07.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 ceph-mon[84949]: mgrmap e1: no daemons active 2023-06-23T01:01:07.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 bash[84884]: d0159ca1176be675e28cc04be2b2786cfec6ef4542f0e931c31cde8b138859a8 2023-06-23T01:01:07.679 INFO:teuthology.orchestra.run.smithi046.stdout:Setting mon public_network to 172.21.0.0/20 2023-06-23T01:01:08.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:07 smithi046 systemd[1]: Started Ceph mon.a for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:01:10.952 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:10 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/714761573' entity='client.admin' 2023-06-23T01:01:11.694 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.523+0000 7fe043dcc700 1 Processor -- start 2023-06-23T01:01:11.694 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.524+0000 7fe043dcc700 1 -- start start 2023-06-23T01:01:11.695 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.524+0000 7fe043dcc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c102a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:11.695 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.524+0000 7fe043dcc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c102f60 con 0x7fe03c102610 2023-06-23T01:01:11.695 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.525+0000 7fe041b68700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c102a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:11.695 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.525+0000 7fe041b68700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c102a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54944/0 (socket says 172.21.15.46:54944) 2023-06-23T01:01:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.525+0000 7fe041b68700 1 -- 172.21.15.46:0/3719839494 learned_addr learned my addr 172.21.15.46:0/3719839494 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.525+0000 7fe041b68700 1 -- 172.21.15.46:0/3719839494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c1030a0 con 0x7fe03c102610 2023-06-23T01:01:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.525+0000 7fe041b68700 1 --2- 172.21.15.46:0/3719839494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c102a20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe0380098e0 tx=0x7fe038009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc86ec7bca07428a server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:11.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.525+0000 7fe040b66700 1 -- 172.21.15.46:0/3719839494 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe038009e80 con 0x7fe03c102610 2023-06-23T01:01:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.525+0000 7fe040b66700 1 -- 172.21.15.46:0/3719839494 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe03800bce0 con 0x7fe03c102610 2023-06-23T01:01:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.525+0000 7fe040b66700 1 -- 172.21.15.46:0/3719839494 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe038011c70 con 0x7fe03c102610 2023-06-23T01:01:11.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.526+0000 7fe043dcc700 1 -- 172.21.15.46:0/3719839494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 msgr2=0x7fe03c102a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.526+0000 7fe043dcc700 1 --2- 172.21.15.46:0/3719839494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c102a20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe0380098e0 tx=0x7fe038009250 comp rx=0 tx=0).stop 2023-06-23T01:01:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.526+0000 7fe043dcc700 1 -- 172.21.15.46:0/3719839494 shutdown_connections 2023-06-23T01:01:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.526+0000 7fe043dcc700 1 --2- 172.21.15.46:0/3719839494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c102a20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:11.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.526+0000 7fe043dcc700 1 -- 172.21.15.46:0/3719839494 >> 172.21.15.46:0/3719839494 conn(0x7fe03c0fbf90 msgr2=0x7fe03c0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:11.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.526+0000 7fe043dcc700 1 -- 172.21.15.46:0/3719839494 shutdown_connections 2023-06-23T01:01:11.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.526+0000 7fe043dcc700 1 -- 172.21.15.46:0/3719839494 wait complete. 2023-06-23T01:01:11.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.526+0000 7fe043dcc700 1 Processor -- start 2023-06-23T01:01:11.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.526+0000 7fe043dcc700 1 -- start start 2023-06-23T01:01:11.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.526+0000 7fe043dcc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:11.704 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.527+0000 7fe043dcc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03c100f30 con 0x7fe03c102610 2023-06-23T01:01:11.704 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.527+0000 7fe041b68700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:11.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.527+0000 7fe041b68700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54954/0 (socket says 172.21.15.46:54954) 2023-06-23T01:01:11.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.527+0000 7fe041b68700 1 -- 172.21.15.46:0/714761573 learned_addr learned my addr 172.21.15.46:0/714761573 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:11.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.527+0000 7fe041b68700 1 -- 172.21.15.46:0/714761573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0380094f0 con 0x7fe03c102610 2023-06-23T01:01:11.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.527+0000 7fe041b68700 1 --2- 172.21.15.46:0/714761573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c113ab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe038011800 tx=0x7fe038009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:11.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.527+0000 7fe0327fc700 1 -- 172.21.15.46:0/714761573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe03801a640 con 0x7fe03c102610 2023-06-23T01:01:11.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.527+0000 7fe0327fc700 1 -- 172.21.15.46:0/714761573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe03801ac20 con 0x7fe03c102610 2023-06-23T01:01:11.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.527+0000 7fe043dcc700 1 -- 172.21.15.46:0/714761573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03c101130 con 0x7fe03c102610 2023-06-23T01:01:11.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.527+0000 7fe0327fc700 1 -- 172.21.15.46:0/714761573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe038022ad0 con 0x7fe03c102610 2023-06-23T01:01:11.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.527+0000 7fe043dcc700 1 -- 172.21.15.46:0/714761573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03c1141b0 con 0x7fe03c102610 2023-06-23T01:01:11.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.528+0000 7fe0327fc700 1 -- 172.21.15.46:0/714761573 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe038022d50 con 0x7fe03c102610 2023-06-23T01:01:11.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.528+0000 7fe0327fc700 1 -- 172.21.15.46:0/714761573 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe03802b730 con 0x7fe03c102610 2023-06-23T01:01:11.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.528+0000 7fe043dcc700 1 -- 172.21.15.46:0/714761573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03c04b970 con 0x7fe03c102610 2023-06-23T01:01:11.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.531+0000 7fe0327fc700 1 -- 172.21.15.46:0/714761573 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fe03801ad90 con 0x7fe03c102610 2023-06-23T01:01:11.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.600+0000 7fe043dcc700 1 -- 172.21.15.46:0/714761573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fe03c101e90 con 0x7fe03c102610 2023-06-23T01:01:11.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.602+0000 7fe0327fc700 1 -- 172.21.15.46:0/714761573 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe03c101e90 con 0x7fe03c102610 2023-06-23T01:01:11.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.603+0000 7fe043dcc700 1 -- 172.21.15.46:0/714761573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 msgr2=0x7fe03c113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:11.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.603+0000 7fe043dcc700 1 --2- 172.21.15.46:0/714761573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c113ab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe038011800 tx=0x7fe038009250 comp rx=0 tx=0).stop 2023-06-23T01:01:11.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.603+0000 7fe043dcc700 1 -- 172.21.15.46:0/714761573 shutdown_connections 2023-06-23T01:01:11.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.603+0000 7fe043dcc700 1 --2- 172.21.15.46:0/714761573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe03c102610 0x7fe03c113ab0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:11.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.603+0000 7fe043dcc700 1 -- 172.21.15.46:0/714761573 >> 172.21.15.46:0/714761573 conn(0x7fe03c0fbf90 msgr2=0x7fe03c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:11.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.603+0000 7fe043dcc700 1 -- 172.21.15.46:0/714761573 shutdown_connections 2023-06-23T01:01:11.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:09.603+0000 7fe043dcc700 1 -- 172.21.15.46:0/714761573 wait complete. 2023-06-23T01:01:11.708 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote config to /etc/ceph/ceph.conf 2023-06-23T01:01:11.708 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-06-23T01:01:11.708 INFO:teuthology.orchestra.run.smithi046.stdout:Creating mgr... 2023-06-23T01:01:11.708 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying port 9283 ... 2023-06-23T01:01:11.845 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.x 2023-06-23T01:01:11.845 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to reset failed state of unit ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.x.service: Unit ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.x.service not loaded. 2023-06-23T01:01:11.944 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-458642ee-1161-11ee-9b2b-001a4aab830c.target.wants/ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-458642ee-1161-11ee-9b2b-001a4aab830c@.service. 2023-06-23T01:01:12.318 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:11 smithi046 systemd[1]: Starting Ceph mgr.x for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:01:12.580 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-06-23T01:01:12.580 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2023-06-23T01:01:12.589 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-06-23T01:01:12.590 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2023-06-23T01:01:12.590 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2023-06-23T01:01:12.590 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to enable service . firewalld.service is not available 2023-06-23T01:01:12.600 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-06-23T01:01:12.600 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2023-06-23T01:01:12.608 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-06-23T01:01:12.609 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2023-06-23T01:01:12.609 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2023-06-23T01:01:12.609 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2023-06-23T01:01:12.609 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr to start... 2023-06-23T01:01:12.609 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr... 2023-06-23T01:01:12.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:12 smithi046 bash[85322]: 20a845f2e7d82ce31497e6281417882c3bced28f688a322e07be57f94becf720 2023-06-23T01:01:12.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:12 smithi046 systemd[1]: Started Ceph mgr.x for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:01:13.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:13 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/1790776385' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-06-23T01:01:13.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:01:13.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2023-06-23T01:01:13.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "458642ee-1161-11ee-9b2b-001a4aab830c", 2023-06-23T01:01:13.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2023-06-23T01:01:13.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-06-23T01:01:13.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-06-23T01:01:13.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-06-23T01:01:13.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:13.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-06-23T01:01:13.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-06-23T01:01:13.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2023-06-23T01:01:13.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:13.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-06-23T01:01:13.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2023-06-23T01:01:13.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:13.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-06-23T01:01:13.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2023-06-23T01:01:13.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:13.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-06-23T01:01:13.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-06-23T01:01:13.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:13.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-06-23T01:01:13.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:13.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-06-23T01:01:13.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-06-23T01:01:13.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-06-23T01:01:13.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-06-23T01:01:13.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-06-23T01:01:13.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-06-23T01:01:13.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:13.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-06-23T01:01:13.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-06-23T01:01:13.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-06-23T01:01:13.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-06-23T01:01:13.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-06-23T01:01:13.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-06-23T01:01:13.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-06-23T01:01:13.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-06-23T01:01:13.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-06-23T01:01:13.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:13.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-06-23T01:01:13.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:13.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-06-23T01:01:13.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-06-23T01:01:13.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:13.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-06-23T01:01:13.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2023-06-23T01:01:13.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-06-23T01:01:13.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2023-06-23T01:01:13.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2023-06-23T01:01:13.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2023-06-23T01:01:13.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2023-06-23T01:01:13.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:13.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-23T01:01:13.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:13.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-06-23T01:01:13.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:13.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2023-06-23T01:00:57.878127+0000", 2023-06-23T01:01:13.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-23T01:01:13.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:13.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-06-23T01:01:13.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2023-06-23T01:01:13.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.097+0000 7fee254c1700 1 Processor -- start 2023-06-23T01:01:13.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.097+0000 7fee254c1700 1 -- start start 2023-06-23T01:01:13.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.097+0000 7fee254c1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20101490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:13.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.097+0000 7fee254c1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee201019d0 con 0x7fee20101080 2023-06-23T01:01:13.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.097+0000 7fee1effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:13.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.097+0000 7fee1effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20101490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54982/0 (socket says 172.21.15.46:54982) 2023-06-23T01:01:13.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.097+0000 7fee1effd700 1 -- 172.21.15.46:0/3757026073 learned_addr learned my addr 172.21.15.46:0/3757026073 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:13.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.097+0000 7fee1effd700 1 -- 172.21.15.46:0/3757026073 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20101b10 con 0x7fee20101080 2023-06-23T01:01:13.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.098+0000 7fee1effd700 1 --2- 172.21.15.46:0/3757026073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20101490 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fee10009e50 tx=0x7fee10009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f46b4d6b910bab server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:13.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.098+0000 7fee1dffb700 1 -- 172.21.15.46:0/3757026073 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee1000e070 con 0x7fee20101080 2023-06-23T01:01:13.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.098+0000 7fee1dffb700 1 -- 172.21.15.46:0/3757026073 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fee10012430 con 0x7fee20101080 2023-06-23T01:01:13.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.098+0000 7fee1dffb700 1 -- 172.21.15.46:0/3757026073 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee1000bea0 con 0x7fee20101080 2023-06-23T01:01:13.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.098+0000 7fee254c1700 1 -- 172.21.15.46:0/3757026073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 msgr2=0x7fee20101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:13.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.098+0000 7fee254c1700 1 --2- 172.21.15.46:0/3757026073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20101490 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fee10009e50 tx=0x7fee10009840 comp rx=0 tx=0).stop 2023-06-23T01:01:13.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.098+0000 7fee254c1700 1 -- 172.21.15.46:0/3757026073 shutdown_connections 2023-06-23T01:01:13.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.098+0000 7fee254c1700 1 --2- 172.21.15.46:0/3757026073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20101490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:13.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.098+0000 7fee254c1700 1 -- 172.21.15.46:0/3757026073 >> 172.21.15.46:0/3757026073 conn(0x7fee200fc800 msgr2=0x7fee200fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:13.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.099+0000 7fee254c1700 1 -- 172.21.15.46:0/3757026073 shutdown_connections 2023-06-23T01:01:13.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.099+0000 7fee254c1700 1 -- 172.21.15.46:0/3757026073 wait complete. 2023-06-23T01:01:13.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.099+0000 7fee254c1700 1 Processor -- start 2023-06-23T01:01:13.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.099+0000 7fee254c1700 1 -- start start 2023-06-23T01:01:13.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.099+0000 7fee254c1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:13.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.099+0000 7fee254c1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee201113e0 con 0x7fee20101080 2023-06-23T01:01:13.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.100+0000 7fee1effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:13.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.100+0000 7fee1effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20110ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54992/0 (socket says 172.21.15.46:54992) 2023-06-23T01:01:13.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.100+0000 7fee1effd700 1 -- 172.21.15.46:0/1790776385 learned_addr learned my addr 172.21.15.46:0/1790776385 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:13.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.100+0000 7fee1effd700 1 -- 172.21.15.46:0/1790776385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee10009ae0 con 0x7fee20101080 2023-06-23T01:01:13.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.100+0000 7fee1effd700 1 --2- 172.21.15.46:0/1790776385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20110ea0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fee1000bc50 tx=0x7fee10009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:13.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.100+0000 7fee177fe700 1 -- 172.21.15.46:0/1790776385 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee1000e030 con 0x7fee20101080 2023-06-23T01:01:13.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.100+0000 7fee177fe700 1 -- 172.21.15.46:0/1790776385 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fee100092e0 con 0x7fee20101080 2023-06-23T01:01:13.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.100+0000 7fee254c1700 1 -- 172.21.15.46:0/1790776385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee201115e0 con 0x7fee20101080 2023-06-23T01:01:13.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.100+0000 7fee254c1700 1 -- 172.21.15.46:0/1790776385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee20112140 con 0x7fee20101080 2023-06-23T01:01:13.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.100+0000 7fee177fe700 1 -- 172.21.15.46:0/1790776385 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee10007580 con 0x7fee20101080 2023-06-23T01:01:13.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.100+0000 7fee177fe700 1 -- 172.21.15.46:0/1790776385 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fee10019070 con 0x7fee20101080 2023-06-23T01:01:13.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.101+0000 7fee177fe700 1 -- 172.21.15.46:0/1790776385 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fee100257f0 con 0x7fee20101080 2023-06-23T01:01:13.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.102+0000 7fee254c1700 1 -- 172.21.15.46:0/1790776385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee000052e0 con 0x7fee20101080 2023-06-23T01:01:13.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.104+0000 7fee177fe700 1 -- 172.21.15.46:0/1790776385 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fee1002a060 con 0x7fee20101080 2023-06-23T01:01:13.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.177+0000 7fee254c1700 1 -- 172.21.15.46:0/1790776385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fee00005860 con 0x7fee20101080 2023-06-23T01:01:13.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.178+0000 7fee177fe700 1 -- 172.21.15.46:0/1790776385 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7fee10020070 con 0x7fee20101080 2023-06-23T01:01:13.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.179+0000 7fee254c1700 1 -- 172.21.15.46:0/1790776385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 msgr2=0x7fee20110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:13.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.179+0000 7fee254c1700 1 --2- 172.21.15.46:0/1790776385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20110ea0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fee1000bc50 tx=0x7fee10009840 comp rx=0 tx=0).stop 2023-06-23T01:01:13.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.179+0000 7fee254c1700 1 -- 172.21.15.46:0/1790776385 shutdown_connections 2023-06-23T01:01:13.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.179+0000 7fee254c1700 1 --2- 172.21.15.46:0/1790776385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee20101080 0x7fee20110ea0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:13.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.179+0000 7fee254c1700 1 -- 172.21.15.46:0/1790776385 >> 172.21.15.46:0/1790776385 conn(0x7fee200fc800 msgr2=0x7fee200709a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:13.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.179+0000 7fee254c1700 1 -- 172.21.15.46:0/1790776385 shutdown_connections 2023-06-23T01:01:13.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:13.179+0000 7fee254c1700 1 -- 172.21.15.46:0/1790776385 wait complete. 2023-06-23T01:01:13.673 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (1/15)... 2023-06-23T01:01:13.959 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:13 smithi046 conmon[85362]: 2023-06-23T01:01:13.793+0000 7f0dd00091c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:01:14.274 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:13 smithi046 conmon[85362]: 2023-06-23T01:01:13.957+0000 7f0dd00091c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:01:14.274 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:14 smithi046 conmon[85362]: 2023-06-23T01:01:14.048+0000 7f0dd00091c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:01:14.568 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:14 smithi046 conmon[85362]: 2023-06-23T01:01:14.271+0000 7f0dd00091c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:01:14.568 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:14 smithi046 conmon[85362]: 2023-06-23T01:01:14.483+0000 7f0dd00091c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:01:15.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:14 smithi046 conmon[85362]: 2023-06-23T01:01:14.656+0000 7f0dd00091c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:01:15.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:14 smithi046 conmon[85362]: 2023-06-23T01:01:14.741+0000 7f0dd00091c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:01:15.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:15 smithi046 conmon[85362]: 2023-06-23T01:01:15.363+0000 7f0dd00091c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:01:16.232 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:15 smithi046 conmon[85362]: 2023-06-23T01:01:15.891+0000 7f0dd00091c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:01:16.232 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:15 smithi046 conmon[85362]: 2023-06-23T01:01:15.989+0000 7f0dd00091c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:01:16.232 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:16 smithi046 conmon[85362]: 2023-06-23T01:01:16.087+0000 7f0dd00091c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:01:16.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:16 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/1136151403' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-06-23T01:01:16.607 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:01:16.607 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2023-06-23T01:01:16.607 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "458642ee-1161-11ee-9b2b-001a4aab830c", 2023-06-23T01:01:16.607 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2023-06-23T01:01:16.608 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-06-23T01:01:16.608 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-06-23T01:01:16.608 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-06-23T01:01:16.608 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:16.608 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-06-23T01:01:16.608 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-06-23T01:01:16.608 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2023-06-23T01:01:16.609 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:16.609 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-06-23T01:01:16.609 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2023-06-23T01:01:16.609 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:16.609 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-06-23T01:01:16.610 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2023-06-23T01:01:16.610 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:16.610 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-06-23T01:01:16.610 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-06-23T01:01:16.610 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:16.610 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-06-23T01:01:16.610 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:16.611 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-06-23T01:01:16.611 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-06-23T01:01:16.611 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-06-23T01:01:16.616 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-06-23T01:01:16.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-06-23T01:01:16.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-06-23T01:01:16.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:16.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-06-23T01:01:16.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-06-23T01:01:16.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-06-23T01:01:16.617 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-06-23T01:01:16.618 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-06-23T01:01:16.618 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-06-23T01:01:16.618 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-06-23T01:01:16.618 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-06-23T01:01:16.618 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-06-23T01:01:16.618 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:16.618 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-06-23T01:01:16.618 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:16.619 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-06-23T01:01:16.619 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-06-23T01:01:16.619 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:16.619 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-06-23T01:01:16.619 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2023-06-23T01:01:16.619 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-06-23T01:01:16.619 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2023-06-23T01:01:16.619 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2023-06-23T01:01:16.620 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2023-06-23T01:01:16.620 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2023-06-23T01:01:16.620 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:16.620 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-23T01:01:16.620 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:16.620 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-06-23T01:01:16.620 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:16.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2023-06-23T01:00:57.878127+0000", 2023-06-23T01:01:16.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-23T01:01:16.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:16.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-06-23T01:01:16.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2023-06-23T01:01:16.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.120+0000 7f92be644700 1 Processor -- start 2023-06-23T01:01:16.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.120+0000 7f92be644700 1 -- start start 2023-06-23T01:01:16.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.121+0000 7f92be644700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8104db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:16.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.121+0000 7f92be644700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b81052f0 con 0x7f92b8070d90 2023-06-23T01:01:16.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.121+0000 7f92b7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:16.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.121+0000 7f92b7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8104db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36618/0 (socket says 172.21.15.46:36618) 2023-06-23T01:01:16.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.121+0000 7f92b7fff700 1 -- 172.21.15.46:0/1293943925 learned_addr learned my addr 172.21.15.46:0/1293943925 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:16.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.121+0000 7f92b7fff700 1 -- 172.21.15.46:0/1293943925 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8105430 con 0x7f92b8070d90 2023-06-23T01:01:16.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.122+0000 7f92b7fff700 1 --2- 172.21.15.46:0/1293943925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8104db0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f92a80098e0 tx=0x7f92a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd69f9649b494a6 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:16.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.122+0000 7f92b6ffd700 1 -- 172.21.15.46:0/1293943925 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92a8009e80 con 0x7f92b8070d90 2023-06-23T01:01:16.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.122+0000 7f92b6ffd700 1 -- 172.21.15.46:0/1293943925 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f92a800bce0 con 0x7f92b8070d90 2023-06-23T01:01:16.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.122+0000 7f92b6ffd700 1 -- 172.21.15.46:0/1293943925 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92a8011c70 con 0x7f92b8070d90 2023-06-23T01:01:16.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.122+0000 7f92be644700 1 -- 172.21.15.46:0/1293943925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 msgr2=0x7f92b8104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:16.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.122+0000 7f92be644700 1 --2- 172.21.15.46:0/1293943925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8104db0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f92a80098e0 tx=0x7f92a8009250 comp rx=0 tx=0).stop 2023-06-23T01:01:16.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.122+0000 7f92be644700 1 -- 172.21.15.46:0/1293943925 shutdown_connections 2023-06-23T01:01:16.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.122+0000 7f92be644700 1 --2- 172.21.15.46:0/1293943925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8104db0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:16.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.122+0000 7f92be644700 1 -- 172.21.15.46:0/1293943925 >> 172.21.15.46:0/1293943925 conn(0x7f92b80fc800 msgr2=0x7f92b80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:16.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.122+0000 7f92be644700 1 -- 172.21.15.46:0/1293943925 shutdown_connections 2023-06-23T01:01:16.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.123+0000 7f92be644700 1 -- 172.21.15.46:0/1293943925 wait complete. 2023-06-23T01:01:16.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.123+0000 7f92be644700 1 Processor -- start 2023-06-23T01:01:16.625 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.123+0000 7f92be644700 1 -- start start 2023-06-23T01:01:16.625 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92be644700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8102830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:16.625 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92be644700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b8100cb0 con 0x7f92b8070d90 2023-06-23T01:01:16.625 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92b7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8102830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:16.625 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92b7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8102830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36630/0 (socket says 172.21.15.46:36630) 2023-06-23T01:01:16.626 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92b7fff700 1 -- 172.21.15.46:0/1136151403 learned_addr learned my addr 172.21.15.46:0/1136151403 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:16.626 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92b7fff700 1 -- 172.21.15.46:0/1136151403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a80094f0 con 0x7f92b8070d90 2023-06-23T01:01:16.626 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92b7fff700 1 --2- 172.21.15.46:0/1136151403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8102830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92a80094c0 tx=0x7f92a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:16.626 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92b4ff9700 1 -- 172.21.15.46:0/1136151403 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92a800bed0 con 0x7f92b8070d90 2023-06-23T01:01:16.626 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92b4ff9700 1 -- 172.21.15.46:0/1136151403 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f92a801aaf0 con 0x7f92b8070d90 2023-06-23T01:01:16.626 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92b4ff9700 1 -- 172.21.15.46:0/1136151403 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92a8022930 con 0x7f92b8070d90 2023-06-23T01:01:16.626 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92be644700 1 -- 172.21.15.46:0/1136151403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b8100eb0 con 0x7f92b8070d90 2023-06-23T01:01:16.627 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.124+0000 7f92be644700 1 -- 172.21.15.46:0/1136151403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b81012c0 con 0x7f92b8070d90 2023-06-23T01:01:16.627 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.125+0000 7f92b4ff9700 1 -- 172.21.15.46:0/1136151403 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f92a8023410 con 0x7f92b8070d90 2023-06-23T01:01:16.627 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.125+0000 7f92b4ff9700 1 -- 172.21.15.46:0/1136151403 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f92a80193c0 con 0x7f92b8070d90 2023-06-23T01:01:16.627 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.126+0000 7f929e7fc700 1 -- 172.21.15.46:0/1136151403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92980052b0 con 0x7f92b8070d90 2023-06-23T01:01:16.627 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.129+0000 7f92b4ff9700 1 -- 172.21.15.46:0/1136151403 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f92a8028060 con 0x7f92b8070d90 2023-06-23T01:01:16.627 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.203+0000 7f929e7fc700 1 -- 172.21.15.46:0/1136151403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9298005830 con 0x7f92b8070d90 2023-06-23T01:01:16.627 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.203+0000 7f92b4ff9700 1 -- 172.21.15.46:0/1136151403 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f92a801e030 con 0x7f92b8070d90 2023-06-23T01:01:16.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.204+0000 7f929e7fc700 1 -- 172.21.15.46:0/1136151403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 msgr2=0x7f92b8102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:16.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.204+0000 7f929e7fc700 1 --2- 172.21.15.46:0/1136151403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8102830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92a80094c0 tx=0x7f92a8009250 comp rx=0 tx=0).stop 2023-06-23T01:01:16.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.204+0000 7f929e7fc700 1 -- 172.21.15.46:0/1136151403 shutdown_connections 2023-06-23T01:01:16.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.204+0000 7f929e7fc700 1 --2- 172.21.15.46:0/1136151403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92b8070d90 0x7f92b8102830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:16.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.204+0000 7f929e7fc700 1 -- 172.21.15.46:0/1136151403 >> 172.21.15.46:0/1136151403 conn(0x7f92b80fc800 msgr2=0x7f92b80fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:16.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.204+0000 7f929e7fc700 1 -- 172.21.15.46:0/1136151403 shutdown_connections 2023-06-23T01:01:16.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:16.204+0000 7f929e7fc700 1 -- 172.21.15.46:0/1136151403 wait complete. 2023-06-23T01:01:16.629 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (2/15)... 2023-06-23T01:01:16.897 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:16 smithi046 conmon[85362]: 2023-06-23T01:01:16.717+0000 7f0dd00091c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:01:17.221 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:16 smithi046 conmon[85362]: 2023-06-23T01:01:16.894+0000 7f0dd00091c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:01:17.221 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:16 smithi046 conmon[85362]: 2023-06-23T01:01:16.989+0000 7f0dd00091c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:01:17.221 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:17 smithi046 conmon[85362]: 2023-06-23T01:01:17.124+0000 7f0dd00091c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:01:17.567 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:17 smithi046 conmon[85362]: 2023-06-23T01:01:17.218+0000 7f0dd00091c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:01:18.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:18 smithi046 conmon[85362]: 2023-06-23T01:01:18.137+0000 7f0dd00091c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:01:18.477 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:18 smithi046 conmon[85362]: 2023-06-23T01:01:18.228+0000 7f0dd00091c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:01:18.477 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:18 smithi046 conmon[85362]: 2023-06-23T01:01:18.319+0000 7f0dd00091c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:01:18.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:18 smithi046 conmon[85362]: 2023-06-23T01:01:18.474+0000 7f0dd00091c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:01:18.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:18 smithi046 conmon[85362]: 2023-06-23T01:01:18.577+0000 7f0dd00091c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:01:19.188 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:18 smithi046 conmon[85362]: 2023-06-23T01:01:18.994+0000 7f0dd00091c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:01:19.189 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:19 smithi046 conmon[85362]: 2023-06-23T01:01:19.091+0000 7f0dd00091c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:01:19.450 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:19 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3917060877' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-06-23T01:01:19.451 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:19 smithi046 conmon[85362]: 2023-06-23T01:01:19.186+0000 7f0dd00091c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:01:19.451 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:19 smithi046 conmon[85362]: 2023-06-23T01:01:19.282+0000 7f0dd00091c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:01:19.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:01:19.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2023-06-23T01:01:19.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "458642ee-1161-11ee-9b2b-001a4aab830c", 2023-06-23T01:01:19.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2023-06-23T01:01:19.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-06-23T01:01:19.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-06-23T01:01:19.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-06-23T01:01:19.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:19.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-06-23T01:01:19.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-06-23T01:01:19.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2023-06-23T01:01:19.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:19.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-06-23T01:01:19.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2023-06-23T01:01:19.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:19.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2023-06-23T01:01:19.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2023-06-23T01:01:19.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:19.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-06-23T01:01:19.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-06-23T01:01:19.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:19.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-06-23T01:01:19.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:19.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-06-23T01:01:19.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-06-23T01:01:19.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-06-23T01:01:19.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-06-23T01:01:19.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-06-23T01:01:19.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-06-23T01:01:19.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:19.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-06-23T01:01:19.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-06-23T01:01:19.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-06-23T01:01:19.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-06-23T01:01:19.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-06-23T01:01:19.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-06-23T01:01:19.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-06-23T01:01:19.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-06-23T01:01:19.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-06-23T01:01:19.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:19.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-06-23T01:01:19.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:19.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-06-23T01:01:19.663 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-06-23T01:01:19.663 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:19.663 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-06-23T01:01:19.663 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2023-06-23T01:01:19.663 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-06-23T01:01:19.663 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2023-06-23T01:01:19.663 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2023-06-23T01:01:19.663 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2023-06-23T01:01:19.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2023-06-23T01:01:19.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:19.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-23T01:01:19.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:19.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-06-23T01:01:19.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:19.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2023-06-23T01:00:57.878127+0000", 2023-06-23T01:01:19.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-23T01:01:19.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:19.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-06-23T01:01:19.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2023-06-23T01:01:19.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.108+0000 7f5da6411700 1 Processor -- start 2023-06-23T01:01:19.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.108+0000 7f5da6411700 1 -- start start 2023-06-23T01:01:19.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.108+0000 7f5da6411700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da0101490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:19.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.108+0000 7f5da6411700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da01019d0 con 0x7f5da0101080 2023-06-23T01:01:19.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.108+0000 7f5d9ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da0101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:19.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.108+0000 7f5d9ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da0101490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36640/0 (socket says 172.21.15.46:36640) 2023-06-23T01:01:19.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.108+0000 7f5d9ffff700 1 -- 172.21.15.46:0/2951273367 learned_addr learned my addr 172.21.15.46:0/2951273367 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:19.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.108+0000 7f5d9ffff700 1 -- 172.21.15.46:0/2951273367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da0101b10 con 0x7f5da0101080 2023-06-23T01:01:19.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.109+0000 7f5d9ffff700 1 --2- 172.21.15.46:0/2951273367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da0101490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5d900098e0 tx=0x7f5d90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121a3b35a7e1d22f server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:19.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.109+0000 7f5d9effd700 1 -- 172.21.15.46:0/2951273367 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d90009e80 con 0x7f5da0101080 2023-06-23T01:01:19.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.109+0000 7f5d9effd700 1 -- 172.21.15.46:0/2951273367 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f5d9000bce0 con 0x7f5da0101080 2023-06-23T01:01:19.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.109+0000 7f5d9effd700 1 -- 172.21.15.46:0/2951273367 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d90011c70 con 0x7f5da0101080 2023-06-23T01:01:19.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.109+0000 7f5da6411700 1 -- 172.21.15.46:0/2951273367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 msgr2=0x7f5da0101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:19.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.109+0000 7f5da6411700 1 --2- 172.21.15.46:0/2951273367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da0101490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5d900098e0 tx=0x7f5d90009250 comp rx=0 tx=0).stop 2023-06-23T01:01:19.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.109+0000 7f5da6411700 1 -- 172.21.15.46:0/2951273367 shutdown_connections 2023-06-23T01:01:19.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.110+0000 7f5da6411700 1 --2- 172.21.15.46:0/2951273367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da0101490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:19.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.110+0000 7f5da6411700 1 -- 172.21.15.46:0/2951273367 >> 172.21.15.46:0/2951273367 conn(0x7f5da00fc800 msgr2=0x7f5da00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:19.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.110+0000 7f5da6411700 1 -- 172.21.15.46:0/2951273367 shutdown_connections 2023-06-23T01:01:19.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.110+0000 7f5da6411700 1 -- 172.21.15.46:0/2951273367 wait complete. 2023-06-23T01:01:19.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.110+0000 7f5da6411700 1 Processor -- start 2023-06-23T01:01:19.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.110+0000 7f5da6411700 1 -- start start 2023-06-23T01:01:19.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.110+0000 7f5da6411700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da010ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:19.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.110+0000 7f5da6411700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da010f100 con 0x7f5da0101080 2023-06-23T01:01:19.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5d9ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da010ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:19.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5d9ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da010ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36656/0 (socket says 172.21.15.46:36656) 2023-06-23T01:01:19.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5d9ffff700 1 -- 172.21.15.46:0/3917060877 learned_addr learned my addr 172.21.15.46:0/3917060877 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:19.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5d9ffff700 1 -- 172.21.15.46:0/3917060877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d900094f0 con 0x7f5da0101080 2023-06-23T01:01:19.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5d9ffff700 1 --2- 172.21.15.46:0/3917060877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da010ebc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5d90011800 tx=0x7f5d90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:19.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5d9cff9700 1 -- 172.21.15.46:0/3917060877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d90006bc0 con 0x7f5da0101080 2023-06-23T01:01:19.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5d9cff9700 1 -- 172.21.15.46:0/3917060877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f5d9001aac0 con 0x7f5da0101080 2023-06-23T01:01:19.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5d9cff9700 1 -- 172.21.15.46:0/3917060877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d90022a50 con 0x7f5da0101080 2023-06-23T01:01:19.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5da6411700 1 -- 172.21.15.46:0/3917060877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da010f300 con 0x7f5da0101080 2023-06-23T01:01:19.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5da6411700 1 -- 172.21.15.46:0/3917060877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da010f710 con 0x7f5da0101080 2023-06-23T01:01:19.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5d9cff9700 1 -- 172.21.15.46:0/3917060877 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5d90022cc0 con 0x7f5da0101080 2023-06-23T01:01:19.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.111+0000 7f5d9cff9700 1 -- 172.21.15.46:0/3917060877 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5d9002b3e0 con 0x7f5da0101080 2023-06-23T01:01:19.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.113+0000 7f5da6411700 1 -- 172.21.15.46:0/3917060877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da004b970 con 0x7f5da0101080 2023-06-23T01:01:19.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.115+0000 7f5d9cff9700 1 -- 172.21.15.46:0/3917060877 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f5d90022400 con 0x7f5da0101080 2023-06-23T01:01:19.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.188+0000 7f5da6411700 1 -- 172.21.15.46:0/3917060877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5da0107e00 con 0x7f5da0101080 2023-06-23T01:01:19.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.189+0000 7f5d9cff9700 1 -- 172.21.15.46:0/3917060877 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (secure 0 0 0) 0x7f5d9001ac30 con 0x7f5da0101080 2023-06-23T01:01:19.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.190+0000 7f5da6411700 1 -- 172.21.15.46:0/3917060877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 msgr2=0x7f5da010ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:19.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.190+0000 7f5da6411700 1 --2- 172.21.15.46:0/3917060877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da010ebc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5d90011800 tx=0x7f5d90009250 comp rx=0 tx=0).stop 2023-06-23T01:01:19.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.190+0000 7f5da6411700 1 -- 172.21.15.46:0/3917060877 shutdown_connections 2023-06-23T01:01:19.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.190+0000 7f5da6411700 1 --2- 172.21.15.46:0/3917060877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5da0101080 0x7f5da010ebc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:19.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.190+0000 7f5da6411700 1 -- 172.21.15.46:0/3917060877 >> 172.21.15.46:0/3917060877 conn(0x7f5da00fc800 msgr2=0x7f5da0105560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:19.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.190+0000 7f5da6411700 1 -- 172.21.15.46:0/3917060877 shutdown_connections 2023-06-23T01:01:19.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:19.190+0000 7f5da6411700 1 -- 172.21.15.46:0/3917060877 wait complete. 2023-06-23T01:01:19.671 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (3/15)... 2023-06-23T01:01:20.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: Activating manager daemon x 2023-06-23T01:01:20.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: mgrmap e2: x(active, starting, since 0.00283104s) 2023-06-23T01:01:20.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: from='mgr.14100 172.21.15.46:0/3227508056' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-06-23T01:01:20.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: from='mgr.14100 172.21.15.46:0/3227508056' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-06-23T01:01:20.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: from='mgr.14100 172.21.15.46:0/3227508056' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-06-23T01:01:20.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: from='mgr.14100 172.21.15.46:0/3227508056' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:01:20.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: from='mgr.14100 172.21.15.46:0/3227508056' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-06-23T01:01:20.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: Manager daemon x is now available 2023-06-23T01:01:20.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: from='mgr.14100 172.21.15.46:0/3227508056' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-06-23T01:01:20.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: from='mgr.14100 172.21.15.46:0/3227508056' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-06-23T01:01:20.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: from='mgr.14100 172.21.15.46:0/3227508056' entity='mgr.x' 2023-06-23T01:01:20.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: from='mgr.14100 172.21.15.46:0/3227508056' entity='mgr.x' 2023-06-23T01:01:20.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:20 smithi046 ceph-mon[84949]: from='mgr.14100 172.21.15.46:0/3227508056' entity='mgr.x' 2023-06-23T01:01:21.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:21 smithi046 ceph-mon[84949]: mgrmap e3: x(active, since 1.00703s) 2023-06-23T01:01:22.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:01:22.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2023-06-23T01:01:22.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "458642ee-1161-11ee-9b2b-001a4aab830c", 2023-06-23T01:01:22.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2023-06-23T01:01:22.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-06-23T01:01:22.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-06-23T01:01:22.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-06-23T01:01:22.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:22.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-06-23T01:01:22.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-06-23T01:01:22.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2023-06-23T01:01:22.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:22.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-06-23T01:01:22.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2023-06-23T01:01:22.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:22.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2023-06-23T01:01:22.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2023-06-23T01:01:22.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:22.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-06-23T01:01:22.913 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-06-23T01:01:22.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:22.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-06-23T01:01:22.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:22.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-06-23T01:01:22.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-06-23T01:01:22.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-06-23T01:01:22.914 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-06-23T01:01:22.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-06-23T01:01:22.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-06-23T01:01:22.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:22.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-06-23T01:01:22.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-06-23T01:01:22.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-06-23T01:01:22.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-06-23T01:01:22.915 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-06-23T01:01:22.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-06-23T01:01:22.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-06-23T01:01:22.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-06-23T01:01:22.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-06-23T01:01:22.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:22.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-06-23T01:01:22.916 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:22.917 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-06-23T01:01:22.917 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-06-23T01:01:22.917 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:22.917 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-06-23T01:01:22.917 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2023-06-23T01:01:22.917 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-06-23T01:01:22.917 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2023-06-23T01:01:22.917 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2023-06-23T01:01:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2023-06-23T01:01:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2023-06-23T01:01:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2023-06-23T01:01:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-23T01:01:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-06-23T01:01:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-06-23T01:01:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2023-06-23T01:00:57.878127+0000", 2023-06-23T01:01:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2023-06-23T01:01:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2023-06-23T01:01:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-06-23T01:01:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2023-06-23T01:01:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.166+0000 7efd357bc700 1 Processor -- start 2023-06-23T01:01:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.166+0000 7efd357bc700 1 -- start start 2023-06-23T01:01:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.166+0000 7efd357bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.166+0000 7efd357bc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd30104190 con 0x7efd30103840 2023-06-23T01:01:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.166+0000 7efd2effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.166+0000 7efd2effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36728/0 (socket says 172.21.15.46:36728) 2023-06-23T01:01:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.166+0000 7efd2effd700 1 -- 172.21.15.46:0/62750326 learned_addr learned my addr 172.21.15.46:0/62750326 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.167+0000 7efd2effd700 1 -- 172.21.15.46:0/62750326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd301042d0 con 0x7efd30103840 2023-06-23T01:01:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.167+0000 7efd2effd700 1 --2- 172.21.15.46:0/62750326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103c50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efd240098e0 tx=0x7efd24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e526bf93114a5d9 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.167+0000 7efd2dffb700 1 -- 172.21.15.46:0/62750326 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd24009e80 con 0x7efd30103840 2023-06-23T01:01:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.167+0000 7efd2dffb700 1 -- 172.21.15.46:0/62750326 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7efd2400bce0 con 0x7efd30103840 2023-06-23T01:01:22.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.167+0000 7efd2dffb700 1 -- 172.21.15.46:0/62750326 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd24011c70 con 0x7efd30103840 2023-06-23T01:01:22.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.168+0000 7efd357bc700 1 -- 172.21.15.46:0/62750326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 msgr2=0x7efd30103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:22.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.168+0000 7efd357bc700 1 --2- 172.21.15.46:0/62750326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103c50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efd240098e0 tx=0x7efd24009250 comp rx=0 tx=0).stop 2023-06-23T01:01:22.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.168+0000 7efd357bc700 1 -- 172.21.15.46:0/62750326 shutdown_connections 2023-06-23T01:01:22.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.168+0000 7efd357bc700 1 --2- 172.21.15.46:0/62750326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103c50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:22.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.168+0000 7efd357bc700 1 -- 172.21.15.46:0/62750326 >> 172.21.15.46:0/62750326 conn(0x7efd300fbfa0 msgr2=0x7efd300fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:22.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.168+0000 7efd357bc700 1 -- 172.21.15.46:0/62750326 shutdown_connections 2023-06-23T01:01:22.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.168+0000 7efd357bc700 1 -- 172.21.15.46:0/62750326 wait complete. 2023-06-23T01:01:22.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.168+0000 7efd357bc700 1 Processor -- start 2023-06-23T01:01:22.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.168+0000 7efd357bc700 1 -- start start 2023-06-23T01:01:22.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.169+0000 7efd357bc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:22.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.169+0000 7efd357bc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd30101500 con 0x7efd30103840 2023-06-23T01:01:22.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.169+0000 7efd2effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:22.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.169+0000 7efd2effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36742/0 (socket says 172.21.15.46:36742) 2023-06-23T01:01:22.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.169+0000 7efd2effd700 1 -- 172.21.15.46:0/3305982461 learned_addr learned my addr 172.21.15.46:0/3305982461 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:22.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.169+0000 7efd2effd700 1 -- 172.21.15.46:0/3305982461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd240094f0 con 0x7efd30103840 2023-06-23T01:01:22.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.169+0000 7efd2effd700 1 --2- 172.21.15.46:0/3305982461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efd24006fd0 tx=0x7efd24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:22.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.169+0000 7efd177fe700 1 -- 172.21.15.46:0/3305982461 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd2400bed0 con 0x7efd30103840 2023-06-23T01:01:22.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.169+0000 7efd177fe700 1 -- 172.21.15.46:0/3305982461 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7efd2401aaf0 con 0x7efd30103840 2023-06-23T01:01:22.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.169+0000 7efd177fe700 1 -- 172.21.15.46:0/3305982461 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd240228d0 con 0x7efd30103840 2023-06-23T01:01:22.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.170+0000 7efd357bc700 1 -- 172.21.15.46:0/3305982461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd30101700 con 0x7efd30103840 2023-06-23T01:01:22.923 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.170+0000 7efd357bc700 1 -- 172.21.15.46:0/3305982461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd30101b10 con 0x7efd30103840 2023-06-23T01:01:22.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.170+0000 7efd177fe700 1 -- 172.21.15.46:0/3305982461 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 3) v1 ==== 43102+0+0 (secure 0 0 0) 0x7efd2401a670 con 0x7efd30103840 2023-06-23T01:01:22.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.170+0000 7efd177fe700 1 --2- 172.21.15.46:0/3305982461 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7efd18036450 0x7efd18038900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:22.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.171+0000 7efd177fe700 1 -- 172.21.15.46:0/3305982461 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7efd24049370 con 0x7efd30103840 2023-06-23T01:01:22.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.171+0000 7efd357bc700 1 -- 172.21.15.46:0/3305982461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3004b970 con 0x7efd30103840 2023-06-23T01:01:22.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.171+0000 7efd2e7fc700 1 --2- 172.21.15.46:0/3305982461 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7efd18036450 0x7efd18038900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:22.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.171+0000 7efd2e7fc700 1 --2- 172.21.15.46:0/3305982461 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7efd18036450 0x7efd18038900 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efd20009990 tx=0x7efd20006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:22.924 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.176+0000 7efd177fe700 1 -- 172.21.15.46:0/3305982461 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7efd2401e070 con 0x7efd30103840 2023-06-23T01:01:22.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.300+0000 7efd177fe700 1 -- 172.21.15.46:0/3305982461 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 4) v1 ==== 43208+0+0 (secure 0 0 0) 0x7efd2401fc80 con 0x7efd30103840 2023-06-23T01:01:22.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.457+0000 7efd357bc700 1 -- 172.21.15.46:0/3305982461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7efd30101dc0 con 0x7efd30103840 2023-06-23T01:01:22.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.457+0000 7efd177fe700 1 -- 172.21.15.46:0/3305982461 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7efd24010380 con 0x7efd30103840 2023-06-23T01:01:22.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.459+0000 7efd357bc700 1 -- 172.21.15.46:0/3305982461 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7efd18036450 msgr2=0x7efd18038900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:22.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.459+0000 7efd357bc700 1 --2- 172.21.15.46:0/3305982461 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7efd18036450 0x7efd18038900 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efd20009990 tx=0x7efd20006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:22.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.459+0000 7efd357bc700 1 -- 172.21.15.46:0/3305982461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 msgr2=0x7efd30103080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:22.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.459+0000 7efd357bc700 1 --2- 172.21.15.46:0/3305982461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efd24006fd0 tx=0x7efd24009250 comp rx=0 tx=0).stop 2023-06-23T01:01:22.925 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.460+0000 7efd357bc700 1 -- 172.21.15.46:0/3305982461 shutdown_connections 2023-06-23T01:01:22.926 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.460+0000 7efd357bc700 1 --2- 172.21.15.46:0/3305982461 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7efd18036450 0x7efd18038900 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:22.926 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.460+0000 7efd357bc700 1 --2- 172.21.15.46:0/3305982461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd30103840 0x7efd30103080 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:22.926 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.460+0000 7efd357bc700 1 -- 172.21.15.46:0/3305982461 >> 172.21.15.46:0/3305982461 conn(0x7efd300fbfa0 msgr2=0x7efd300fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:22.926 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.460+0000 7efd357bc700 1 -- 172.21.15.46:0/3305982461 shutdown_connections 2023-06-23T01:01:22.926 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:22.460+0000 7efd357bc700 1 -- 172.21.15.46:0/3305982461 wait complete. 2023-06-23T01:01:22.926 INFO:teuthology.orchestra.run.smithi046.stdout:mgr is available 2023-06-23T01:01:23.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:23 smithi046 ceph-mon[84949]: mgrmap e4: x(active, since 3s) 2023-06-23T01:01:23.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:23 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3305982461' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-06-23T01:01:23.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:01:23.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [global] 2023-06-23T01:01:23.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout fsid = 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:01:23.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-06-23T01:01:23.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-06-23T01:01:23.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-06-23T01:01:23.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-06-23T01:01:23.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:01:23.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [mgr] 2023-06-23T01:01:23.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-06-23T01:01:23.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2023-06-23T01:01:23.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [osd] 2023-06-23T01:01:23.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2023-06-23T01:01:23.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-06-23T01:01:23.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-06-23T01:01:23.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-06-23T01:01:23.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-06-23T01:01:23.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-06-23T01:01:23.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.377+0000 7fdf80608700 1 Processor -- start 2023-06-23T01:01:23.954 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.377+0000 7fdf80608700 1 -- start start 2023-06-23T01:01:23.954 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.377+0000 7fdf80608700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf78104e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:23.954 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.377+0000 7fdf80608700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf78105370 con 0x7fdf78070d10 2023-06-23T01:01:23.954 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.378+0000 7fdf7e3a4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf78104e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:23.954 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.378+0000 7fdf7e3a4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf78104e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36756/0 (socket says 172.21.15.46:36756) 2023-06-23T01:01:23.954 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.378+0000 7fdf7e3a4700 1 -- 172.21.15.46:0/3699346942 learned_addr learned my addr 172.21.15.46:0/3699346942 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:23.954 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.378+0000 7fdf7e3a4700 1 -- 172.21.15.46:0/3699346942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf781054b0 con 0x7fdf78070d10 2023-06-23T01:01:23.954 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.378+0000 7fdf7e3a4700 1 --2- 172.21.15.46:0/3699346942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf78104e30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009e50 tx=0x7fdf74009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d342dd5d489e3eb8 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:23.955 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.378+0000 7fdf7d3a2700 1 -- 172.21.15.46:0/3699346942 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf7400e070 con 0x7fdf78070d10 2023-06-23T01:01:23.955 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.378+0000 7fdf7d3a2700 1 -- 172.21.15.46:0/3699346942 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fdf74012430 con 0x7fdf78070d10 2023-06-23T01:01:23.955 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.378+0000 7fdf7d3a2700 1 -- 172.21.15.46:0/3699346942 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf7400bea0 con 0x7fdf78070d10 2023-06-23T01:01:23.955 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.379+0000 7fdf80608700 1 -- 172.21.15.46:0/3699346942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 msgr2=0x7fdf78104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:23.955 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.379+0000 7fdf80608700 1 --2- 172.21.15.46:0/3699346942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf78104e30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009e50 tx=0x7fdf74009840 comp rx=0 tx=0).stop 2023-06-23T01:01:23.955 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.379+0000 7fdf80608700 1 -- 172.21.15.46:0/3699346942 shutdown_connections 2023-06-23T01:01:23.955 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.379+0000 7fdf80608700 1 --2- 172.21.15.46:0/3699346942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf78104e30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:23.955 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.379+0000 7fdf80608700 1 -- 172.21.15.46:0/3699346942 >> 172.21.15.46:0/3699346942 conn(0x7fdf780fc800 msgr2=0x7fdf780fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:23.956 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.379+0000 7fdf80608700 1 -- 172.21.15.46:0/3699346942 shutdown_connections 2023-06-23T01:01:23.956 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.379+0000 7fdf80608700 1 -- 172.21.15.46:0/3699346942 wait complete. 2023-06-23T01:01:23.956 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.379+0000 7fdf80608700 1 Processor -- start 2023-06-23T01:01:23.956 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.379+0000 7fdf80608700 1 -- start start 2023-06-23T01:01:23.956 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf80608700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf7810ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:23.956 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf80608700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7810f280 con 0x7fdf78070d10 2023-06-23T01:01:23.957 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf7e3a4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf7810ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:23.957 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf7e3a4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf7810ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36766/0 (socket says 172.21.15.46:36766) 2023-06-23T01:01:23.957 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf7e3a4700 1 -- 172.21.15.46:0/1475409459 learned_addr learned my addr 172.21.15.46:0/1475409459 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:23.957 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf7e3a4700 1 -- 172.21.15.46:0/1475409459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf74009ae0 con 0x7fdf78070d10 2023-06-23T01:01:23.957 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf7e3a4700 1 --2- 172.21.15.46:0/1475409459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf7810ed40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdf74012cc0 tx=0x7fdf74009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:23.957 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf6effd700 1 -- 172.21.15.46:0/1475409459 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf7400e040 con 0x7fdf78070d10 2023-06-23T01:01:23.957 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf6effd700 1 -- 172.21.15.46:0/1475409459 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fdf7400bd30 con 0x7fdf78070d10 2023-06-23T01:01:23.957 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf6effd700 1 -- 172.21.15.46:0/1475409459 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf7401c540 con 0x7fdf78070d10 2023-06-23T01:01:23.958 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf80608700 1 -- 172.21.15.46:0/1475409459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7810f480 con 0x7fdf78070d10 2023-06-23T01:01:23.958 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.380+0000 7fdf80608700 1 -- 172.21.15.46:0/1475409459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7810ffe0 con 0x7fdf78070d10 2023-06-23T01:01:23.958 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.381+0000 7fdf6effd700 1 -- 172.21.15.46:0/1475409459 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 43208+0+0 (secure 0 0 0) 0x7fdf7401c7b0 con 0x7fdf78070d10 2023-06-23T01:01:23.958 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.381+0000 7fdf6effd700 1 --2- 172.21.15.46:0/1475409459 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fdf6403a9c0 0x7fdf6403ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:23.958 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.382+0000 7fdf6effd700 1 -- 172.21.15.46:0/1475409459 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdf7404a140 con 0x7fdf78070d10 2023-06-23T01:01:23.958 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.382+0000 7fdf80608700 1 -- 172.21.15.46:0/1475409459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7805f060 con 0x7fdf78070d10 2023-06-23T01:01:23.958 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.382+0000 7fdf7dba3700 1 --2- 172.21.15.46:0/1475409459 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fdf6403a9c0 0x7fdf6403ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:23.959 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.382+0000 7fdf7dba3700 1 --2- 172.21.15.46:0/1475409459 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fdf6403a9c0 0x7fdf6403ce70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf68009990 tx=0x7fdf68006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:23.959 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.386+0000 7fdf6effd700 1 -- 172.21.15.46:0/1475409459 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fdf74017080 con 0x7fdf78070d10 2023-06-23T01:01:23.959 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.579+0000 7fdf80608700 1 -- 172.21.15.46:0/1475409459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fdf78100f70 con 0x7fdf78070d10 2023-06-23T01:01:23.959 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.580+0000 7fdf6effd700 1 -- 172.21.15.46:0/1475409459 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7fdf74036df0 con 0x7fdf78070d10 2023-06-23T01:01:23.959 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.582+0000 7fdf80608700 1 -- 172.21.15.46:0/1475409459 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fdf6403a9c0 msgr2=0x7fdf6403ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:23.959 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.582+0000 7fdf80608700 1 --2- 172.21.15.46:0/1475409459 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fdf6403a9c0 0x7fdf6403ce70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf68009990 tx=0x7fdf68006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:23.959 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.582+0000 7fdf80608700 1 -- 172.21.15.46:0/1475409459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 msgr2=0x7fdf7810ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:23.959 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.582+0000 7fdf80608700 1 --2- 172.21.15.46:0/1475409459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf7810ed40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdf74012cc0 tx=0x7fdf74009840 comp rx=0 tx=0).stop 2023-06-23T01:01:23.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.582+0000 7fdf80608700 1 -- 172.21.15.46:0/1475409459 shutdown_connections 2023-06-23T01:01:23.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.582+0000 7fdf80608700 1 --2- 172.21.15.46:0/1475409459 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fdf6403a9c0 0x7fdf6403ce70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:23.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.582+0000 7fdf80608700 1 --2- 172.21.15.46:0/1475409459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf78070d10 0x7fdf7810ed40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:23.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.582+0000 7fdf80608700 1 -- 172.21.15.46:0/1475409459 >> 172.21.15.46:0/1475409459 conn(0x7fdf780fc800 msgr2=0x7fdf781027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:23.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.582+0000 7fdf80608700 1 -- 172.21.15.46:0/1475409459 shutdown_connections 2023-06-23T01:01:23.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:23.582+0000 7fdf80608700 1 -- 172.21.15.46:0/1475409459 wait complete. 2023-06-23T01:01:23.961 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling cephadm module... 2023-06-23T01:01:24.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:24 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/1475409459' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-06-23T01:01:25.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:25 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/441630121' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-06-23T01:01:25.569 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:25 smithi046 conmon[85362]: ignoring --setuser ceph since I am not root 2023-06-23T01:01:25.569 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:25 smithi046 conmon[85362]: ignoring --setgroup ceph since I am not root 2023-06-23T01:01:25.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.450+0000 7fabee26a700 1 Processor -- start 2023-06-23T01:01:25.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.450+0000 7fabee26a700 1 -- start start 2023-06-23T01:01:25.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.450+0000 7fabee26a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8076160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:25.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.450+0000 7fabee26a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe8078070 con 0x7fabe8077c60 2023-06-23T01:01:25.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.450+0000 7fabe7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8076160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:25.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.450+0000 7fabe7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8076160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37216/0 (socket says 172.21.15.46:37216) 2023-06-23T01:01:25.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.450+0000 7fabe7fff700 1 -- 172.21.15.46:0/736508390 learned_addr learned my addr 172.21.15.46:0/736508390 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:25.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.450+0000 7fabe7fff700 1 -- 172.21.15.46:0/736508390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe80766a0 con 0x7fabe8077c60 2023-06-23T01:01:25.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.450+0000 7fabe7fff700 1 --2- 172.21.15.46:0/736508390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8076160 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fabd80098e0 tx=0x7fabd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8beda3508d45c1d8 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:25.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.451+0000 7fabe6ffd700 1 -- 172.21.15.46:0/736508390 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabd8009e80 con 0x7fabe8077c60 2023-06-23T01:01:25.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.451+0000 7fabe6ffd700 1 -- 172.21.15.46:0/736508390 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fabd800bce0 con 0x7fabe8077c60 2023-06-23T01:01:25.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.451+0000 7fabe6ffd700 1 -- 172.21.15.46:0/736508390 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabd8011c70 con 0x7fabe8077c60 2023-06-23T01:01:25.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.451+0000 7fabee26a700 1 -- 172.21.15.46:0/736508390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 msgr2=0x7fabe8076160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:25.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.451+0000 7fabee26a700 1 --2- 172.21.15.46:0/736508390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8076160 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fabd80098e0 tx=0x7fabd8009250 comp rx=0 tx=0).stop 2023-06-23T01:01:25.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.451+0000 7fabee26a700 1 -- 172.21.15.46:0/736508390 shutdown_connections 2023-06-23T01:01:25.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.451+0000 7fabee26a700 1 --2- 172.21.15.46:0/736508390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8076160 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:25.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.451+0000 7fabee26a700 1 -- 172.21.15.46:0/736508390 >> 172.21.15.46:0/736508390 conn(0x7fabe80fd900 msgr2=0x7fabe80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:25.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.451+0000 7fabee26a700 1 -- 172.21.15.46:0/736508390 shutdown_connections 2023-06-23T01:01:25.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.451+0000 7fabee26a700 1 -- 172.21.15.46:0/736508390 wait complete. 2023-06-23T01:01:25.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.452+0000 7fabee26a700 1 Processor -- start 2023-06-23T01:01:25.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.452+0000 7fabee26a700 1 -- start start 2023-06-23T01:01:25.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.452+0000 7fabee26a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8112f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:25.796 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.452+0000 7fabee26a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe8113450 con 0x7fabe8077c60 2023-06-23T01:01:25.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.452+0000 7fabe7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8112f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:25.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.452+0000 7fabe7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8112f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37232/0 (socket says 172.21.15.46:37232) 2023-06-23T01:01:25.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.452+0000 7fabe7fff700 1 -- 172.21.15.46:0/441630121 learned_addr learned my addr 172.21.15.46:0/441630121 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:25.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.452+0000 7fabe7fff700 1 -- 172.21.15.46:0/441630121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd80094f0 con 0x7fabe8077c60 2023-06-23T01:01:25.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.453+0000 7fabe7fff700 1 --2- 172.21.15.46:0/441630121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8112f10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fabd8011800 tx=0x7fabd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:25.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.453+0000 7fabe4ff9700 1 -- 172.21.15.46:0/441630121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabd801a670 con 0x7fabe8077c60 2023-06-23T01:01:25.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.453+0000 7fabe4ff9700 1 -- 172.21.15.46:0/441630121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fabd801ac50 con 0x7fabe8077c60 2023-06-23T01:01:25.797 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.453+0000 7fabee26a700 1 -- 172.21.15.46:0/441630121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe8113650 con 0x7fabe8077c60 2023-06-23T01:01:25.798 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.453+0000 7fabe4ff9700 1 -- 172.21.15.46:0/441630121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fabd8022990 con 0x7fabe8077c60 2023-06-23T01:01:25.798 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.453+0000 7fabee26a700 1 -- 172.21.15.46:0/441630121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe81142f0 con 0x7fabe8077c60 2023-06-23T01:01:25.798 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.454+0000 7fabe4ff9700 1 -- 172.21.15.46:0/441630121 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 43208+0+0 (secure 0 0 0) 0x7fabd8022c50 con 0x7fabe8077c60 2023-06-23T01:01:25.798 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.454+0000 7fabee26a700 1 -- 172.21.15.46:0/441630121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe805f060 con 0x7fabe8077c60 2023-06-23T01:01:25.798 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.454+0000 7fabe4ff9700 1 --2- 172.21.15.46:0/441630121 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fabd0036520 0x7fabd00389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:25.798 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.454+0000 7fabe4ff9700 1 -- 172.21.15.46:0/441630121 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fabd80496e0 con 0x7fabe8077c60 2023-06-23T01:01:25.798 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.454+0000 7fabe77fe700 1 --2- 172.21.15.46:0/441630121 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fabd0036520 0x7fabd00389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:25.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.454+0000 7fabe77fe700 1 --2- 172.21.15.46:0/441630121 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fabd0036520 0x7fabd00389d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009990 tx=0x7fabdc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:25.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.458+0000 7fabe4ff9700 1 -- 172.21.15.46:0/441630121 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fabd8010b60 con 0x7fabe8077c60 2023-06-23T01:01:25.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:24.709+0000 7fabee26a700 1 -- 172.21.15.46:0/441630121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fabe804b970 con 0x7fabe8077c60 2023-06-23T01:01:25.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.304+0000 7fabe4ff9700 1 -- 172.21.15.46:0/441630121 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 5) v1 ==== 43219+0+0 (secure 0 0 0) 0x7fabd8016d00 con 0x7fabe8077c60 2023-06-23T01:01:25.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.306+0000 7fabe4ff9700 1 -- 172.21.15.46:0/441630121 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fabd801f450 con 0x7fabe8077c60 2023-06-23T01:01:25.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.308+0000 7fabee26a700 1 -- 172.21.15.46:0/441630121 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fabd0036520 msgr2=0x7fabd00389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:25.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.308+0000 7fabee26a700 1 --2- 172.21.15.46:0/441630121 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fabd0036520 0x7fabd00389d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009990 tx=0x7fabdc006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:25.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.308+0000 7fabee26a700 1 -- 172.21.15.46:0/441630121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 msgr2=0x7fabe8112f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:25.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.308+0000 7fabee26a700 1 --2- 172.21.15.46:0/441630121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8112f10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fabd8011800 tx=0x7fabd8009250 comp rx=0 tx=0).stop 2023-06-23T01:01:25.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.308+0000 7fabee26a700 1 -- 172.21.15.46:0/441630121 shutdown_connections 2023-06-23T01:01:25.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.308+0000 7fabee26a700 1 --2- 172.21.15.46:0/441630121 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fabd0036520 0x7fabd00389d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:25.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.308+0000 7fabee26a700 1 --2- 172.21.15.46:0/441630121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabe8077c60 0x7fabe8112f10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:25.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.308+0000 7fabee26a700 1 -- 172.21.15.46:0/441630121 >> 172.21.15.46:0/441630121 conn(0x7fabe80fd900 msgr2=0x7fabe80fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:25.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.309+0000 7fabee26a700 1 -- 172.21.15.46:0/441630121 shutdown_connections 2023-06-23T01:01:25.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:25.309+0000 7fabee26a700 1 -- 172.21.15.46:0/441630121 wait complete. 2023-06-23T01:01:26.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:26 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/441630121' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-06-23T01:01:26.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:26 smithi046 ceph-mon[84949]: mgrmap e5: x(active, since 6s) 2023-06-23T01:01:26.898 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:26 smithi046 conmon[85362]: 2023-06-23T01:01:26.632+0000 7feb78b2f1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:01:26.898 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:26 smithi046 conmon[85362]: 2023-06-23T01:01:26.801+0000 7feb78b2f1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:01:26.898 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:26 smithi046 conmon[85362]: 2023-06-23T01:01:26.895+0000 7feb78b2f1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:01:26.930 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2023-06-23T01:01:26.935 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-06-23T01:01:26.935 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2023-06-23T01:01:26.936 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-06-23T01:01:26.937 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-06-23T01:01:26.937 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2023-06-23T01:01:26.937 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.262+0000 7f0dcc282700 1 Processor -- start 2023-06-23T01:01:26.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.263+0000 7f0dcc282700 1 -- start start 2023-06-23T01:01:26.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.263+0000 7f0dcc282700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4104d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:26.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.263+0000 7f0dcc282700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc41052d0 con 0x7f0dc4070db0 2023-06-23T01:01:26.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.263+0000 7f0dca01e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4104d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:26.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.263+0000 7f0dca01e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4104d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37252/0 (socket says 172.21.15.46:37252) 2023-06-23T01:01:26.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.263+0000 7f0dca01e700 1 -- 172.21.15.46:0/2692430510 learned_addr learned my addr 172.21.15.46:0/2692430510 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:26.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.263+0000 7f0dca01e700 1 -- 172.21.15.46:0/2692430510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc4105410 con 0x7f0dc4070db0 2023-06-23T01:01:26.938 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.264+0000 7f0dca01e700 1 --2- 172.21.15.46:0/2692430510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4104d90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00098e0 tx=0x7f0dc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d010ed5cd07f48 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:26.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.264+0000 7f0dc901c700 1 -- 172.21.15.46:0/2692430510 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dc0009e80 con 0x7f0dc4070db0 2023-06-23T01:01:26.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.264+0000 7f0dc901c700 1 -- 172.21.15.46:0/2692430510 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f0dc000bce0 con 0x7f0dc4070db0 2023-06-23T01:01:26.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.264+0000 7f0dc901c700 1 -- 172.21.15.46:0/2692430510 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dc0011c70 con 0x7f0dc4070db0 2023-06-23T01:01:26.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.264+0000 7f0dcc282700 1 -- 172.21.15.46:0/2692430510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 msgr2=0x7f0dc4104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:26.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.264+0000 7f0dcc282700 1 --2- 172.21.15.46:0/2692430510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4104d90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00098e0 tx=0x7f0dc0009250 comp rx=0 tx=0).stop 2023-06-23T01:01:26.939 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.264+0000 7f0dcc282700 1 -- 172.21.15.46:0/2692430510 shutdown_connections 2023-06-23T01:01:26.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.264+0000 7f0dcc282700 1 --2- 172.21.15.46:0/2692430510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4104d90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:26.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.264+0000 7f0dcc282700 1 -- 172.21.15.46:0/2692430510 >> 172.21.15.46:0/2692430510 conn(0x7f0dc40fc800 msgr2=0x7f0dc40fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:26.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.265+0000 7f0dcc282700 1 -- 172.21.15.46:0/2692430510 shutdown_connections 2023-06-23T01:01:26.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.265+0000 7f0dcc282700 1 -- 172.21.15.46:0/2692430510 wait complete. 2023-06-23T01:01:26.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.265+0000 7f0dcc282700 1 Processor -- start 2023-06-23T01:01:26.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.265+0000 7f0dcc282700 1 -- start start 2023-06-23T01:01:26.940 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.265+0000 7f0dcc282700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:26.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.265+0000 7f0dcc282700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc41113e0 con 0x7f0dc4070db0 2023-06-23T01:01:26.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.265+0000 7f0dca01e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:26.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.266+0000 7f0dca01e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4110ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37268/0 (socket says 172.21.15.46:37268) 2023-06-23T01:01:26.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.266+0000 7f0dca01e700 1 -- 172.21.15.46:0/993545560 learned_addr learned my addr 172.21.15.46:0/993545560 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:26.941 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.266+0000 7f0dca01e700 1 -- 172.21.15.46:0/993545560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc00094f0 con 0x7f0dc4070db0 2023-06-23T01:01:26.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.266+0000 7f0dca01e700 1 --2- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4110ea0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0011800 tx=0x7f0dc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:26.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.266+0000 7f0db6ffd700 1 -- 172.21.15.46:0/993545560 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dc001a640 con 0x7f0dc4070db0 2023-06-23T01:01:26.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.266+0000 7f0db6ffd700 1 -- 172.21.15.46:0/993545560 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f0dc001ac20 con 0x7f0dc4070db0 2023-06-23T01:01:26.942 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.266+0000 7f0dcc282700 1 -- 172.21.15.46:0/993545560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc41115e0 con 0x7f0dc4070db0 2023-06-23T01:01:26.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.266+0000 7f0db6ffd700 1 -- 172.21.15.46:0/993545560 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dc0022ad0 con 0x7f0dc4070db0 2023-06-23T01:01:26.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.266+0000 7f0dcc282700 1 -- 172.21.15.46:0/993545560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc4112140 con 0x7f0dc4070db0 2023-06-23T01:01:26.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.267+0000 7f0dcc282700 1 -- 172.21.15.46:0/993545560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc41119f0 con 0x7f0dc4070db0 2023-06-23T01:01:26.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.267+0000 7f0db6ffd700 1 -- 172.21.15.46:0/993545560 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f0dc0022400 con 0x7f0dc4070db0 2023-06-23T01:01:26.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.268+0000 7f0db6ffd700 1 --2- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7f0db00365c0 0x7f0db0038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:26.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.268+0000 7f0db6ffd700 1 -- 172.21.15.46:0/993545560 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0dc004a440 con 0x7f0dc4070db0 2023-06-23T01:01:26.943 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.268+0000 7f0dc981d700 1 -- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7f0db00365c0 msgr2=0x7f0db0038a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3291477171 2023-06-23T01:01:26.944 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.268+0000 7f0dc981d700 1 --2- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7f0db00365c0 0x7f0db0038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:01:26.945 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.272+0000 7f0db6ffd700 1 -- 172.21.15.46:0/993545560 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f0dc41119f0 con 0x7f0dc4070db0 2023-06-23T01:01:26.945 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.469+0000 7f0dc981d700 1 -- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7f0db00365c0 msgr2=0x7f0db0038a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3291477171 2023-06-23T01:01:26.945 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.469+0000 7f0dc981d700 1 --2- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7f0db00365c0 0x7f0db0038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-23T01:01:26.945 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.521+0000 7f0dcc282700 1 -- 172.21.15.46:0/993545560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f0dc41119f0 con 0x7f0dc4070db0 2023-06-23T01:01:26.946 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.522+0000 7f0db6ffd700 1 -- 172.21.15.46:0/993545560 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f0dc41119f0 con 0x7f0dc4070db0 2023-06-23T01:01:26.946 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.524+0000 7f0dcc282700 1 -- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7f0db00365c0 msgr2=0x7f0db0038a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-23T01:01:26.946 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.524+0000 7f0dcc282700 1 --2- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7f0db00365c0 0x7f0db0038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:26.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.524+0000 7f0dcc282700 1 -- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 msgr2=0x7f0dc4110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:26.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.524+0000 7f0dcc282700 1 --2- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4110ea0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0011800 tx=0x7f0dc0009250 comp rx=0 tx=0).stop 2023-06-23T01:01:26.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.524+0000 7f0dcc282700 1 -- 172.21.15.46:0/993545560 shutdown_connections 2023-06-23T01:01:26.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.524+0000 7f0dcc282700 1 --2- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7f0db00365c0 0x7f0db0038a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:26.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.524+0000 7f0dcc282700 1 --2- 172.21.15.46:0/993545560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc4070db0 0x7f0dc4110ea0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:26.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.524+0000 7f0dcc282700 1 -- 172.21.15.46:0/993545560 >> 172.21.15.46:0/993545560 conn(0x7f0dc40fc800 msgr2=0x7f0dc40fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:26.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.524+0000 7f0dcc282700 1 -- 172.21.15.46:0/993545560 shutdown_connections 2023-06-23T01:01:26.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:26.524+0000 7f0dcc282700 1 -- 172.21.15.46:0/993545560 wait complete. 2023-06-23T01:01:26.948 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for the mgr to restart... 2023-06-23T01:01:26.949 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr epoch 5... 2023-06-23T01:01:27.174 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:27 smithi046 conmon[85362]: 2023-06-23T01:01:27.132+0000 7feb78b2f1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:01:27.524 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:27 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/993545560' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-06-23T01:01:27.524 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:27 smithi046 conmon[85362]: 2023-06-23T01:01:27.352+0000 7feb78b2f1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:01:27.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:27 smithi046 conmon[85362]: 2023-06-23T01:01:27.521+0000 7feb78b2f1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:01:27.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:27 smithi046 conmon[85362]: 2023-06-23T01:01:27.604+0000 7feb78b2f1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:01:28.568 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:28 smithi046 conmon[85362]: 2023-06-23T01:01:28.215+0000 7feb78b2f1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:01:29.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:28 smithi046 conmon[85362]: 2023-06-23T01:01:28.721+0000 7feb78b2f1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:01:29.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:28 smithi046 conmon[85362]: 2023-06-23T01:01:28.817+0000 7feb78b2f1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:01:29.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:28 smithi046 conmon[85362]: 2023-06-23T01:01:28.913+0000 7feb78b2f1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:01:29.805 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:29 smithi046 conmon[85362]: 2023-06-23T01:01:29.529+0000 7feb78b2f1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:01:29.805 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:29 smithi046 conmon[85362]: 2023-06-23T01:01:29.706+0000 7feb78b2f1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:01:30.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:29 smithi046 conmon[85362]: 2023-06-23T01:01:29.803+0000 7feb78b2f1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:01:30.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:29 smithi046 conmon[85362]: 2023-06-23T01:01:29.937+0000 7feb78b2f1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:01:30.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:30 smithi046 conmon[85362]: 2023-06-23T01:01:30.032+0000 7feb78b2f1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:01:31.295 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:30 smithi046 conmon[85362]: 2023-06-23T01:01:30.954+0000 7feb78b2f1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:01:31.295 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:31 smithi046 conmon[85362]: 2023-06-23T01:01:31.046+0000 7feb78b2f1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:01:31.295 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:31 smithi046 conmon[85362]: 2023-06-23T01:01:31.137+0000 7feb78b2f1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:01:31.568 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:31 smithi046 conmon[85362]: 2023-06-23T01:01:31.292+0000 7feb78b2f1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:01:31.568 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:31 smithi046 conmon[85362]: 2023-06-23T01:01:31.395+0000 7feb78b2f1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:01:32.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:31 smithi046 conmon[85362]: 2023-06-23T01:01:31.802+0000 7feb78b2f1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:01:32.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:31 smithi046 conmon[85362]: 2023-06-23T01:01:31.896+0000 7feb78b2f1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:01:32.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:31 smithi046 conmon[85362]: 2023-06-23T01:01:31.988+0000 7feb78b2f1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:01:32.568 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:32 smithi046 conmon[85362]: 2023-06-23T01:01:32.081+0000 7feb78b2f1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:01:32.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: Active manager daemon x restarted 2023-06-23T01:01:32.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: Activating manager daemon x 2023-06-23T01:01:32.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: osdmap e2: 0 total, 0 up, 0 in 2023-06-23T01:01:32.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: mgrmap e6: x(active, starting, since 0.0030271s) 2023-06-23T01:01:32.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:01:32.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-06-23T01:01:32.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-06-23T01:01:32.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-06-23T01:01:32.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-06-23T01:01:32.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: Manager daemon x is now available 2023-06-23T01:01:32.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:32.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:32.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:01:32.571 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:01:32.571 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:32 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:01:33.521 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:33 smithi046 ceph-mon[84949]: Found migration_current of "None". Setting to last migration. 2023-06-23T01:01:33.521 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:33 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-06-23T01:01:33.521 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:33 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-06-23T01:01:33.521 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:33 smithi046 ceph-mon[84949]: mgrmap e7: x(active, since 1.00506s) 2023-06-23T01:01:33.591 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2023-06-23T01:01:33.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-06-23T01:01:33.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "initialized": true 2023-06-23T01:01:33.592 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2023-06-23T01:01:33.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.455+0000 7fd2d709d700 1 Processor -- start 2023-06-23T01:01:33.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.455+0000 7fd2d709d700 1 -- start start 2023-06-23T01:01:33.593 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.455+0000 7fd2d709d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0104db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:33.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.455+0000 7fd2d709d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d01052f0 con 0x7fd2d0070d90 2023-06-23T01:01:33.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.455+0000 7fd2d4e39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:33.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.455+0000 7fd2d4e39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0104db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37282/0 (socket says 172.21.15.46:37282) 2023-06-23T01:01:33.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.455+0000 7fd2d4e39700 1 -- 172.21.15.46:0/3128947210 learned_addr learned my addr 172.21.15.46:0/3128947210 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:33.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.456+0000 7fd2d4e39700 1 -- 172.21.15.46:0/3128947210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d0105430 con 0x7fd2d0070d90 2023-06-23T01:01:33.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.456+0000 7fd2d4e39700 1 --2- 172.21.15.46:0/3128947210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0104db0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40098e0 tx=0x7fd2c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac8701d6f5343fe server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:33.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.456+0000 7fd2cf7fe700 1 -- 172.21.15.46:0/3128947210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd2c4009e80 con 0x7fd2d0070d90 2023-06-23T01:01:33.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.456+0000 7fd2cf7fe700 1 -- 172.21.15.46:0/3128947210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd2c400bce0 con 0x7fd2d0070d90 2023-06-23T01:01:33.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.456+0000 7fd2cf7fe700 1 -- 172.21.15.46:0/3128947210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd2c4011c70 con 0x7fd2d0070d90 2023-06-23T01:01:33.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.457+0000 7fd2d709d700 1 -- 172.21.15.46:0/3128947210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 msgr2=0x7fd2d0104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:33.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.457+0000 7fd2d709d700 1 --2- 172.21.15.46:0/3128947210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0104db0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40098e0 tx=0x7fd2c4009250 comp rx=0 tx=0).stop 2023-06-23T01:01:33.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.457+0000 7fd2d709d700 1 -- 172.21.15.46:0/3128947210 shutdown_connections 2023-06-23T01:01:33.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.457+0000 7fd2d709d700 1 --2- 172.21.15.46:0/3128947210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0104db0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:33.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.457+0000 7fd2d709d700 1 -- 172.21.15.46:0/3128947210 >> 172.21.15.46:0/3128947210 conn(0x7fd2d00fc820 msgr2=0x7fd2d00fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:33.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.457+0000 7fd2d709d700 1 -- 172.21.15.46:0/3128947210 shutdown_connections 2023-06-23T01:01:33.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.457+0000 7fd2d709d700 1 -- 172.21.15.46:0/3128947210 wait complete. 2023-06-23T01:01:33.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.458+0000 7fd2d709d700 1 Processor -- start 2023-06-23T01:01:33.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.458+0000 7fd2d709d700 1 -- start start 2023-06-23T01:01:33.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.458+0000 7fd2d709d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0102830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:33.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.458+0000 7fd2d709d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d0100cb0 con 0x7fd2d0070d90 2023-06-23T01:01:33.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.458+0000 7fd2d4e39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0102830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:33.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.458+0000 7fd2d4e39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0102830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37298/0 (socket says 172.21.15.46:37298) 2023-06-23T01:01:33.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.458+0000 7fd2d4e39700 1 -- 172.21.15.46:0/1023242307 learned_addr learned my addr 172.21.15.46:0/1023242307 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:33.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.458+0000 7fd2d4e39700 1 -- 172.21.15.46:0/1023242307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c40094f0 con 0x7fd2d0070d90 2023-06-23T01:01:33.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.458+0000 7fd2d4e39700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0102830 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4011800 tx=0x7fd2c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:33.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.459+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd2c401a640 con 0x7fd2d0070d90 2023-06-23T01:01:33.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.459+0000 7fd2d709d700 1 -- 172.21.15.46:0/1023242307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d0100eb0 con 0x7fd2d0070d90 2023-06-23T01:01:33.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.459+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd2c401ac20 con 0x7fd2d0070d90 2023-06-23T01:01:33.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.459+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd2c4022ad0 con 0x7fd2d0070d90 2023-06-23T01:01:33.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.459+0000 7fd2d709d700 1 -- 172.21.15.46:0/1023242307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d01012c0 con 0x7fd2d0070d90 2023-06-23T01:01:33.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.460+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 43219+0+0 (secure 0 0 0) 0x7fd2c4022d50 con 0x7fd2d0070d90 2023-06-23T01:01:33.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.460+0000 7fd2cd7fa700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 0x7fd2c0038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:33.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.460+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 --> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2c0039120 con 0x7fd2c0036560 2023-06-23T01:01:33.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.460+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd2c404a4d0 con 0x7fd2d0070d90 2023-06-23T01:01:33.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.460+0000 7fd2cffff700 1 -- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 msgr2=0x7fd2c0038a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3291477171 2023-06-23T01:01:33.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.460+0000 7fd2cffff700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 0x7fd2c0038a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:01:33.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.661+0000 7fd2cffff700 1 -- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 msgr2=0x7fd2c0038a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3291477171 2023-06-23T01:01:33.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:27.661+0000 7fd2cffff700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 0x7fd2c0038a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-23T01:01:33.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:28.062+0000 7fd2cffff700 1 -- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 msgr2=0x7fd2c0038a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3291477171 2023-06-23T01:01:33.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:28.062+0000 7fd2cffff700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 0x7fd2c0038a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-23T01:01:33.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:28.863+0000 7fd2cffff700 1 -- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 msgr2=0x7fd2c0038a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3291477171 2023-06-23T01:01:33.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:28.863+0000 7fd2cffff700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 0x7fd2c0038a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-23T01:01:33.602 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:30.464+0000 7fd2cffff700 1 -- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 msgr2=0x7fd2c0038a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/3291477171 2023-06-23T01:01:33.602 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:30.464+0000 7fd2cffff700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 0x7fd2c0038a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-06-23T01:01:33.602 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:32.097+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 6) v1 ==== 42986+0+0 (secure 0 0 0) 0x7fd2c402b3e0 con 0x7fd2d0070d90 2023-06-23T01:01:33.602 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:32.097+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 msgr2=0x7fd2c0038a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-23T01:01:33.603 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:32.097+0000 7fd2cd7fa700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 0x7fd2c0038a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:33.603 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.098+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 7) v1 ==== 43113+0+0 (secure 0 0 0) 0x7fd2c400e950 con 0x7fd2d0070d90 2023-06-23T01:01:33.603 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.098+0000 7fd2cd7fa700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fd2c0039e80 0x7fd2c003c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:33.603 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.098+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 --> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2c0039120 con 0x7fd2c0039e80 2023-06-23T01:01:33.603 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.098+0000 7fd2cffff700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fd2c0039e80 0x7fd2c003c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:33.603 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.099+0000 7fd2cffff700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fd2c0039e80 0x7fd2c003c260 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0039d0 tx=0x7fd2bc009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:33.603 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.100+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 <== mgr.14120 v2:172.21.15.46:6800/2331157358 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fd2c0039120 con 0x7fd2c0039e80 2023-06-23T01:01:33.604 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.110+0000 7fd2d709d700 1 -- 172.21.15.46:0/1023242307 --> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd2d005f060 con 0x7fd2c0039e80 2023-06-23T01:01:33.604 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.110+0000 7fd2cd7fa700 1 -- 172.21.15.46:0/1023242307 <== mgr.14120 v2:172.21.15.46:6800/2331157358 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd2d005f060 con 0x7fd2c0039e80 2023-06-23T01:01:33.604 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.111+0000 7fd2d709d700 1 -- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fd2c0039e80 msgr2=0x7fd2c003c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:33.604 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.111+0000 7fd2d709d700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fd2c0039e80 0x7fd2c003c260 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0039d0 tx=0x7fd2bc009350 comp rx=0 tx=0).stop 2023-06-23T01:01:33.604 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.111+0000 7fd2d709d700 1 -- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 msgr2=0x7fd2d0102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:33.604 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.111+0000 7fd2d709d700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0102830 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4011800 tx=0x7fd2c4009250 comp rx=0 tx=0).stop 2023-06-23T01:01:33.605 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.111+0000 7fd2d709d700 1 -- 172.21.15.46:0/1023242307 shutdown_connections 2023-06-23T01:01:33.605 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.111+0000 7fd2d709d700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fd2c0039e80 0x7fd2c003c260 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:33.605 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.111+0000 7fd2d709d700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d0070d90 0x7fd2d0102830 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:33.605 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.111+0000 7fd2d709d700 1 --2- 172.21.15.46:0/1023242307 >> [v2:172.21.15.46:6800/3291477171,v1:172.21.15.46:6801/3291477171] conn(0x7fd2c0036560 0x7fd2c0038a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:33.605 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.111+0000 7fd2d709d700 1 -- 172.21.15.46:0/1023242307 >> 172.21.15.46:0/1023242307 conn(0x7fd2d00fc820 msgr2=0x7fd2d00fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:33.605 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.111+0000 7fd2d709d700 1 -- 172.21.15.46:0/1023242307 shutdown_connections 2023-06-23T01:01:33.605 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:33.111+0000 7fd2d709d700 1 -- 172.21.15.46:0/1023242307 wait complete. 2023-06-23T01:01:33.606 INFO:teuthology.orchestra.run.smithi046.stdout:mgr epoch 5 is available 2023-06-23T01:01:33.606 INFO:teuthology.orchestra.run.smithi046.stdout:Setting orchestrator backend to cephadm... 2023-06-23T01:01:34.465 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:34 smithi046 ceph-mon[84949]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-06-23T01:01:34.465 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:34 smithi046 ceph-mon[84949]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-06-23T01:01:34.465 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:34 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:34.466 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:34 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:01:34.627 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.024+0000 7f70a9450700 1 Processor -- start 2023-06-23T01:01:34.627 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.024+0000 7f70a9450700 1 -- start start 2023-06-23T01:01:34.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.024+0000 7f70a9450700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4104d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:34.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.024+0000 7f70a9450700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a41052b0 con 0x7f70a4070dc0 2023-06-23T01:01:34.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.024+0000 7f70a2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4104d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:34.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.024+0000 7f70a2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4104d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37374/0 (socket says 172.21.15.46:37374) 2023-06-23T01:01:34.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.024+0000 7f70a2ffd700 1 -- 172.21.15.46:0/4196195649 learned_addr learned my addr 172.21.15.46:0/4196195649 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:34.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.025+0000 7f70a2ffd700 1 -- 172.21.15.46:0/4196195649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a41053f0 con 0x7f70a4070dc0 2023-06-23T01:01:34.628 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.025+0000 7f70a2ffd700 1 --2- 172.21.15.46:0/4196195649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4104d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f70940098e0 tx=0x7f7094009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78c048514cb934 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:34.629 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.025+0000 7f70a1ffb700 1 -- 172.21.15.46:0/4196195649 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7094009e80 con 0x7f70a4070dc0 2023-06-23T01:01:34.629 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.025+0000 7f70a1ffb700 1 -- 172.21.15.46:0/4196195649 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f709400bce0 con 0x7f70a4070dc0 2023-06-23T01:01:34.629 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.025+0000 7f70a1ffb700 1 -- 172.21.15.46:0/4196195649 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7094011c70 con 0x7f70a4070dc0 2023-06-23T01:01:34.629 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.025+0000 7f70a9450700 1 -- 172.21.15.46:0/4196195649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 msgr2=0x7f70a4104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:34.629 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.025+0000 7f70a9450700 1 --2- 172.21.15.46:0/4196195649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4104d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f70940098e0 tx=0x7f7094009250 comp rx=0 tx=0).stop 2023-06-23T01:01:34.629 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.026+0000 7f70a9450700 1 -- 172.21.15.46:0/4196195649 shutdown_connections 2023-06-23T01:01:34.629 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.026+0000 7f70a9450700 1 --2- 172.21.15.46:0/4196195649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4104d70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:34.629 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.026+0000 7f70a9450700 1 -- 172.21.15.46:0/4196195649 >> 172.21.15.46:0/4196195649 conn(0x7f70a40fc7e0 msgr2=0x7f70a40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:34.630 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.026+0000 7f70a9450700 1 -- 172.21.15.46:0/4196195649 shutdown_connections 2023-06-23T01:01:34.630 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.026+0000 7f70a9450700 1 -- 172.21.15.46:0/4196195649 wait complete. 2023-06-23T01:01:34.630 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.026+0000 7f70a9450700 1 Processor -- start 2023-06-23T01:01:34.630 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.026+0000 7f70a9450700 1 -- start start 2023-06-23T01:01:34.630 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.026+0000 7f70a9450700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4110db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:34.630 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.026+0000 7f70a9450700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a41112f0 con 0x7f70a4070dc0 2023-06-23T01:01:34.630 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.027+0000 7f70a2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4110db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.027+0000 7f70a2ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4110db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37388/0 (socket says 172.21.15.46:37388) 2023-06-23T01:01:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.027+0000 7f70a2ffd700 1 -- 172.21.15.46:0/2863927722 learned_addr learned my addr 172.21.15.46:0/2863927722 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.027+0000 7f70a2ffd700 1 -- 172.21.15.46:0/2863927722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70940094f0 con 0x7f70a4070dc0 2023-06-23T01:01:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.027+0000 7f70a2ffd700 1 --2- 172.21.15.46:0/2863927722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4110db0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7094011800 tx=0x7f7094009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.027+0000 7f708b7fe700 1 -- 172.21.15.46:0/2863927722 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f709401a640 con 0x7f70a4070dc0 2023-06-23T01:01:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.027+0000 7f708b7fe700 1 -- 172.21.15.46:0/2863927722 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f709401ac20 con 0x7f70a4070dc0 2023-06-23T01:01:34.631 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.027+0000 7f70a9450700 1 -- 172.21.15.46:0/2863927722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a41114f0 con 0x7f70a4070dc0 2023-06-23T01:01:34.632 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.027+0000 7f708b7fe700 1 -- 172.21.15.46:0/2863927722 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70940229e0 con 0x7f70a4070dc0 2023-06-23T01:01:34.632 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.027+0000 7f70a9450700 1 -- 172.21.15.46:0/2863927722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a41120e0 con 0x7f70a4070dc0 2023-06-23T01:01:34.632 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.028+0000 7f708b7fe700 1 -- 172.21.15.46:0/2863927722 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 7) v1 ==== 43113+0+0 (secure 0 0 0) 0x7f7094022c50 con 0x7f70a4070dc0 2023-06-23T01:01:34.632 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.028+0000 7f708b7fe700 1 --2- 172.21.15.46:0/2863927722 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f708c0364f0 0x7f708c0389a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:34.632 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.028+0000 7f70a9450700 1 -- 172.21.15.46:0/2863927722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a405f060 con 0x7f70a4070dc0 2023-06-23T01:01:34.632 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.028+0000 7f708b7fe700 1 -- 172.21.15.46:0/2863927722 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f709404a340 con 0x7f70a4070dc0 2023-06-23T01:01:34.632 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.029+0000 7f70a27fc700 1 --2- 172.21.15.46:0/2863927722 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f708c0364f0 0x7f708c0389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:34.633 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.029+0000 7f70a27fc700 1 --2- 172.21.15.46:0/2863927722 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f708c0364f0 0x7f708c0389a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7098009990 tx=0x7f7098006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:34.633 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.033+0000 7f708b7fe700 1 -- 172.21.15.46:0/2863927722 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f7094022400 con 0x7f70a4070dc0 2023-06-23T01:01:34.633 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.241+0000 7f70a9450700 1 -- 172.21.15.46:0/2863927722 --> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f70a40fdba0 con 0x7f708c0364f0 2023-06-23T01:01:34.633 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.246+0000 7f708b7fe700 1 -- 172.21.15.46:0/2863927722 <== mgr.14120 v2:172.21.15.46:6800/2331157358 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f70a40fdba0 con 0x7f708c0364f0 2023-06-23T01:01:34.633 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.248+0000 7f70a9450700 1 -- 172.21.15.46:0/2863927722 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f708c0364f0 msgr2=0x7f708c0389a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:34.633 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.248+0000 7f70a9450700 1 --2- 172.21.15.46:0/2863927722 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f708c0364f0 0x7f708c0389a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7098009990 tx=0x7f7098006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:34.633 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.248+0000 7f70a9450700 1 -- 172.21.15.46:0/2863927722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 msgr2=0x7f70a4110db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:34.634 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.248+0000 7f70a9450700 1 --2- 172.21.15.46:0/2863927722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4110db0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7094011800 tx=0x7f7094009250 comp rx=0 tx=0).stop 2023-06-23T01:01:34.634 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.249+0000 7f70a9450700 1 -- 172.21.15.46:0/2863927722 shutdown_connections 2023-06-23T01:01:34.634 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.249+0000 7f70a9450700 1 --2- 172.21.15.46:0/2863927722 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f708c0364f0 0x7f708c0389a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:34.634 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.249+0000 7f70a9450700 1 --2- 172.21.15.46:0/2863927722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a4070dc0 0x7f70a4110db0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:34.634 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.249+0000 7f70a9450700 1 -- 172.21.15.46:0/2863927722 >> 172.21.15.46:0/2863927722 conn(0x7f70a40fc7e0 msgr2=0x7f70a40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:34.634 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.249+0000 7f70a9450700 1 -- 172.21.15.46:0/2863927722 shutdown_connections 2023-06-23T01:01:34.634 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:34.249+0000 7f70a9450700 1 -- 172.21.15.46:0/2863927722 wait complete. 2023-06-23T01:01:35.499 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:35 smithi046 ceph-mon[84949]: [23/Jun/2023:01:01:33] ENGINE Bus STARTING 2023-06-23T01:01:35.499 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:35 smithi046 ceph-mon[84949]: [23/Jun/2023:01:01:33] ENGINE Serving on https://172.21.15.46:7150 2023-06-23T01:01:35.499 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:35 smithi046 ceph-mon[84949]: [23/Jun/2023:01:01:33] ENGINE Bus STARTED 2023-06-23T01:01:35.500 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:35 smithi046 ceph-mon[84949]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:01:35.500 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:35 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:35.501 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:35 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:01:35.501 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:35 smithi046 ceph-mon[84949]: mgrmap e8: x(active, since 2s) 2023-06-23T01:01:35.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout value unchanged 2023-06-23T01:01:35.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.059+0000 7f066b247700 1 Processor -- start 2023-06-23T01:01:35.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.059+0000 7f066b247700 1 -- start start 2023-06-23T01:01:35.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.059+0000 7f066b247700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664104d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:35.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.059+0000 7f066b247700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06641052d0 con 0x7f0664070db0 2023-06-23T01:01:35.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.059+0000 7f0668fe3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664104d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:35.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.060+0000 7f0668fe3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664104d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41256/0 (socket says 172.21.15.46:41256) 2023-06-23T01:01:35.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.060+0000 7f0668fe3700 1 -- 172.21.15.46:0/2995178844 learned_addr learned my addr 172.21.15.46:0/2995178844 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:35.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.060+0000 7f0668fe3700 1 -- 172.21.15.46:0/2995178844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0664105410 con 0x7f0664070db0 2023-06-23T01:01:35.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.060+0000 7f0668fe3700 1 --2- 172.21.15.46:0/2995178844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664104d90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f06580098e0 tx=0x7f0658009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79384e48da8210b server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:35.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.060+0000 7f06637fe700 1 -- 172.21.15.46:0/2995178844 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0658009e80 con 0x7f0664070db0 2023-06-23T01:01:35.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.060+0000 7f06637fe700 1 -- 172.21.15.46:0/2995178844 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f065800bce0 con 0x7f0664070db0 2023-06-23T01:01:35.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.060+0000 7f06637fe700 1 -- 172.21.15.46:0/2995178844 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0658011c70 con 0x7f0664070db0 2023-06-23T01:01:35.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.061+0000 7f066b247700 1 -- 172.21.15.46:0/2995178844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 msgr2=0x7f0664104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:35.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.061+0000 7f066b247700 1 --2- 172.21.15.46:0/2995178844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664104d90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f06580098e0 tx=0x7f0658009250 comp rx=0 tx=0).stop 2023-06-23T01:01:35.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.061+0000 7f066b247700 1 -- 172.21.15.46:0/2995178844 shutdown_connections 2023-06-23T01:01:35.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.061+0000 7f066b247700 1 --2- 172.21.15.46:0/2995178844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664104d90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:35.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.061+0000 7f066b247700 1 -- 172.21.15.46:0/2995178844 >> 172.21.15.46:0/2995178844 conn(0x7f06640fc800 msgr2=0x7f06640fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:35.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.061+0000 7f066b247700 1 -- 172.21.15.46:0/2995178844 shutdown_connections 2023-06-23T01:01:35.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.061+0000 7f066b247700 1 -- 172.21.15.46:0/2995178844 wait complete. 2023-06-23T01:01:35.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.061+0000 7f066b247700 1 Processor -- start 2023-06-23T01:01:35.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.061+0000 7f066b247700 1 -- start start 2023-06-23T01:01:35.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.062+0000 7f066b247700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:35.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.062+0000 7f066b247700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06641113e0 con 0x7f0664070db0 2023-06-23T01:01:35.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.062+0000 7f0668fe3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:35.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.062+0000 7f0668fe3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664110ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41268/0 (socket says 172.21.15.46:41268) 2023-06-23T01:01:35.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.062+0000 7f0668fe3700 1 -- 172.21.15.46:0/350318816 learned_addr learned my addr 172.21.15.46:0/350318816 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:35.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.062+0000 7f0668fe3700 1 -- 172.21.15.46:0/350318816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06580094f0 con 0x7f0664070db0 2023-06-23T01:01:35.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.062+0000 7f0668fe3700 1 --2- 172.21.15.46:0/350318816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664110ea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f06580094c0 tx=0x7f0658009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:35.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.062+0000 7f06617fa700 1 -- 172.21.15.46:0/350318816 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f065800bed0 con 0x7f0664070db0 2023-06-23T01:01:35.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.063+0000 7f06617fa700 1 -- 172.21.15.46:0/350318816 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f065801aaf0 con 0x7f0664070db0 2023-06-23T01:01:35.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.063+0000 7f06617fa700 1 -- 172.21.15.46:0/350318816 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0658022930 con 0x7f0664070db0 2023-06-23T01:01:35.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.063+0000 7f066b247700 1 -- 172.21.15.46:0/350318816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06641115e0 con 0x7f0664070db0 2023-06-23T01:01:35.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.063+0000 7f066b247700 1 -- 172.21.15.46:0/350318816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0664112140 con 0x7f0664070db0 2023-06-23T01:01:35.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.063+0000 7f06617fa700 1 -- 172.21.15.46:0/350318816 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f0658022b20 con 0x7f0664070db0 2023-06-23T01:01:35.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.063+0000 7f06617fa700 1 --2- 172.21.15.46:0/350318816 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f06540365c0 0x7f0654038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:35.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.064+0000 7f06617fa700 1 -- 172.21.15.46:0/350318816 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f06580498e0 con 0x7f0664070db0 2023-06-23T01:01:35.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.064+0000 7f066b247700 1 -- 172.21.15.46:0/350318816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06641119f0 con 0x7f0664070db0 2023-06-23T01:01:35.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.064+0000 7f0663fff700 1 --2- 172.21.15.46:0/350318816 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f06540365c0 0x7f0654038a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:35.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.064+0000 7f0663fff700 1 --2- 172.21.15.46:0/350318816 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f06540365c0 0x7f0654038a70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0650009990 tx=0x7f0650006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:35.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.069+0000 7f06617fa700 1 -- 172.21.15.46:0/350318816 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f06641119f0 con 0x7f0664070db0 2023-06-23T01:01:35.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.265+0000 7f066b247700 1 -- 172.21.15.46:0/350318816 --> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f06641119f0 con 0x7f06540365c0 2023-06-23T01:01:35.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.266+0000 7f06617fa700 1 -- 172.21.15.46:0/350318816 <== mgr.14120 v2:172.21.15.46:6800/2331157358 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f06641119f0 con 0x7f06540365c0 2023-06-23T01:01:35.672 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.268+0000 7f066b247700 1 -- 172.21.15.46:0/350318816 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f06540365c0 msgr2=0x7f0654038a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:35.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.268+0000 7f066b247700 1 --2- 172.21.15.46:0/350318816 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f06540365c0 0x7f0654038a70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0650009990 tx=0x7f0650006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:35.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.268+0000 7f066b247700 1 -- 172.21.15.46:0/350318816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 msgr2=0x7f0664110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:35.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.268+0000 7f066b247700 1 --2- 172.21.15.46:0/350318816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664110ea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f06580094c0 tx=0x7f0658009250 comp rx=0 tx=0).stop 2023-06-23T01:01:35.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.268+0000 7f066b247700 1 -- 172.21.15.46:0/350318816 shutdown_connections 2023-06-23T01:01:35.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.268+0000 7f066b247700 1 --2- 172.21.15.46:0/350318816 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f06540365c0 0x7f0654038a70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:35.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.268+0000 7f066b247700 1 --2- 172.21.15.46:0/350318816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0664070db0 0x7f0664110ea0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:35.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.268+0000 7f066b247700 1 -- 172.21.15.46:0/350318816 >> 172.21.15.46:0/350318816 conn(0x7f06640fc800 msgr2=0x7f06640fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:35.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.268+0000 7f066b247700 1 -- 172.21.15.46:0/350318816 shutdown_connections 2023-06-23T01:01:35.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:35.269+0000 7f066b247700 1 -- 172.21.15.46:0/350318816 wait complete. 2023-06-23T01:01:35.674 INFO:teuthology.orchestra.run.smithi046.stdout:Generating ssh key... 2023-06-23T01:01:36.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:36 smithi046 ceph-mon[84949]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:01:37.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:37 smithi046 ceph-mon[84949]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:01:37.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:37 smithi046 ceph-mon[84949]: Generating ssh key... 2023-06-23T01:01:37.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:37 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:37.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:37 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:37.319 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: Generating public/private rsa key pair. 2023-06-23T01:01:37.319 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: Your identification has been saved in /tmp/tmpl7oat46b/key. 2023-06-23T01:01:37.319 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: Your public key has been saved in /tmp/tmpl7oat46b/key.pub. 2023-06-23T01:01:37.319 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: The key fingerprint is: 2023-06-23T01:01:37.319 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: SHA256:qH2BawpOqKEcQyYvB/fb7OrBbOelejGNjdMiWKpbduc ceph-458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:01:37.319 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: The key's randomart image is: 2023-06-23T01:01:37.319 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: +---[RSA 3072]----+ 2023-06-23T01:01:37.320 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: | | 2023-06-23T01:01:37.320 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: | | 2023-06-23T01:01:37.320 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: | | 2023-06-23T01:01:37.320 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: | . o | 2023-06-23T01:01:37.320 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: |oo.+ o*S | 2023-06-23T01:01:37.321 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: |+=o+.oB.+. | 2023-06-23T01:01:37.322 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: |++B O.*=o | 2023-06-23T01:01:37.322 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: |=Oo+ %.+ | 2023-06-23T01:01:37.322 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: |+o..*=E | 2023-06-23T01:01:37.323 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:37 smithi046 conmon[85362]: +----[SHA256]-----+ 2023-06-23T01:01:37.506 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.108+0000 7f3e57873700 1 Processor -- start 2023-06-23T01:01:37.506 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.108+0000 7f3e57873700 1 -- start start 2023-06-23T01:01:37.506 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.108+0000 7f3e57873700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50101490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:37.506 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.108+0000 7f3e57873700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e501019d0 con 0x7f3e50101080 2023-06-23T01:01:37.506 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.109+0000 7f3e5560f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:37.507 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.109+0000 7f3e5560f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50101490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41284/0 (socket says 172.21.15.46:41284) 2023-06-23T01:01:37.507 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.109+0000 7f3e5560f700 1 -- 172.21.15.46:0/3871865154 learned_addr learned my addr 172.21.15.46:0/3871865154 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:37.507 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.109+0000 7f3e5560f700 1 -- 172.21.15.46:0/3871865154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e50101b10 con 0x7f3e50101080 2023-06-23T01:01:37.507 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.109+0000 7f3e5560f700 1 --2- 172.21.15.46:0/3871865154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50101490 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3e480098e0 tx=0x7f3e48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd917106d918f4b server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:37.507 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.109+0000 7f3e4ffff700 1 -- 172.21.15.46:0/3871865154 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e48009e80 con 0x7f3e50101080 2023-06-23T01:01:37.507 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.109+0000 7f3e4ffff700 1 -- 172.21.15.46:0/3871865154 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f3e4800bce0 con 0x7f3e50101080 2023-06-23T01:01:37.507 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.109+0000 7f3e4ffff700 1 -- 172.21.15.46:0/3871865154 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e48011c70 con 0x7f3e50101080 2023-06-23T01:01:37.508 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.110+0000 7f3e57873700 1 -- 172.21.15.46:0/3871865154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 msgr2=0x7f3e50101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:37.508 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.110+0000 7f3e57873700 1 --2- 172.21.15.46:0/3871865154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50101490 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3e480098e0 tx=0x7f3e48009250 comp rx=0 tx=0).stop 2023-06-23T01:01:37.508 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.110+0000 7f3e57873700 1 -- 172.21.15.46:0/3871865154 shutdown_connections 2023-06-23T01:01:37.508 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.110+0000 7f3e57873700 1 --2- 172.21.15.46:0/3871865154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50101490 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:37.508 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.110+0000 7f3e57873700 1 -- 172.21.15.46:0/3871865154 >> 172.21.15.46:0/3871865154 conn(0x7f3e500fc800 msgr2=0x7f3e500fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:37.508 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.110+0000 7f3e57873700 1 -- 172.21.15.46:0/3871865154 shutdown_connections 2023-06-23T01:01:37.508 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.110+0000 7f3e57873700 1 -- 172.21.15.46:0/3871865154 wait complete. 2023-06-23T01:01:37.509 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.110+0000 7f3e57873700 1 Processor -- start 2023-06-23T01:01:37.509 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.110+0000 7f3e57873700 1 -- start start 2023-06-23T01:01:37.509 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.111+0000 7f3e57873700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50107430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:37.509 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.111+0000 7f3e57873700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e501058b0 con 0x7f3e50101080 2023-06-23T01:01:37.509 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.111+0000 7f3e5560f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50107430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:37.509 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.111+0000 7f3e5560f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50107430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41288/0 (socket says 172.21.15.46:41288) 2023-06-23T01:01:37.509 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.111+0000 7f3e5560f700 1 -- 172.21.15.46:0/1348273396 learned_addr learned my addr 172.21.15.46:0/1348273396 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:37.509 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.111+0000 7f3e5560f700 1 -- 172.21.15.46:0/1348273396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e480094f0 con 0x7f3e50101080 2023-06-23T01:01:37.510 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.111+0000 7f3e5560f700 1 --2- 172.21.15.46:0/1348273396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50107430 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3e48006fd0 tx=0x7f3e48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:37.510 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.111+0000 7f3e4dffb700 1 -- 172.21.15.46:0/1348273396 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e4800bed0 con 0x7f3e50101080 2023-06-23T01:01:37.510 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.111+0000 7f3e4dffb700 1 -- 172.21.15.46:0/1348273396 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f3e4801aaf0 con 0x7f3e50101080 2023-06-23T01:01:37.510 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.111+0000 7f3e57873700 1 -- 172.21.15.46:0/1348273396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e50105ab0 con 0x7f3e50101080 2023-06-23T01:01:37.510 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.111+0000 7f3e4dffb700 1 -- 172.21.15.46:0/1348273396 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e480228d0 con 0x7f3e50101080 2023-06-23T01:01:37.510 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.112+0000 7f3e57873700 1 -- 172.21.15.46:0/1348273396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e50105ec0 con 0x7f3e50101080 2023-06-23T01:01:37.510 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.112+0000 7f3e4dffb700 1 -- 172.21.15.46:0/1348273396 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f3e4801a670 con 0x7f3e50101080 2023-06-23T01:01:37.511 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.113+0000 7f3e4dffb700 1 --2- 172.21.15.46:0/1348273396 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e380365c0 0x7f3e38038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:37.511 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.113+0000 7f3e57873700 1 -- 172.21.15.46:0/1348273396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5004b970 con 0x7f3e50101080 2023-06-23T01:01:37.511 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.113+0000 7f3e54e0e700 1 --2- 172.21.15.46:0/1348273396 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e380365c0 0x7f3e38038a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:37.511 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.113+0000 7f3e4dffb700 1 -- 172.21.15.46:0/1348273396 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3e48049950 con 0x7f3e50101080 2023-06-23T01:01:37.511 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.113+0000 7f3e54e0e700 1 --2- 172.21.15.46:0/1348273396 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e380365c0 0x7f3e38038a70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009990 tx=0x7f3e44006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:37.511 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.117+0000 7f3e4dffb700 1 -- 172.21.15.46:0/1348273396 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f3e48022b60 con 0x7f3e50101080 2023-06-23T01:01:37.511 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:36.313+0000 7f3e57873700 1 -- 172.21.15.46:0/1348273396 --> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e50106700 con 0x7f3e380365c0 2023-06-23T01:01:37.511 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:37.055+0000 7f3e4dffb700 1 -- 172.21.15.46:0/1348273396 <== mgr.14120 v2:172.21.15.46:6800/2331157358 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3e50106700 con 0x7f3e380365c0 2023-06-23T01:01:37.512 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:37.057+0000 7f3e57873700 1 -- 172.21.15.46:0/1348273396 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e380365c0 msgr2=0x7f3e38038a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:37.512 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:37.057+0000 7f3e57873700 1 --2- 172.21.15.46:0/1348273396 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e380365c0 0x7f3e38038a70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009990 tx=0x7f3e44006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:37.512 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:37.057+0000 7f3e57873700 1 -- 172.21.15.46:0/1348273396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 msgr2=0x7f3e50107430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:37.512 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:37.057+0000 7f3e57873700 1 --2- 172.21.15.46:0/1348273396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50107430 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3e48006fd0 tx=0x7f3e48009250 comp rx=0 tx=0).stop 2023-06-23T01:01:37.512 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:37.058+0000 7f3e57873700 1 -- 172.21.15.46:0/1348273396 shutdown_connections 2023-06-23T01:01:37.512 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:37.058+0000 7f3e57873700 1 --2- 172.21.15.46:0/1348273396 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e380365c0 0x7f3e38038a70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:37.512 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:37.058+0000 7f3e57873700 1 --2- 172.21.15.46:0/1348273396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e50101080 0x7f3e50107430 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:37.513 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:37.058+0000 7f3e57873700 1 -- 172.21.15.46:0/1348273396 >> 172.21.15.46:0/1348273396 conn(0x7f3e500fc800 msgr2=0x7f3e500fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:37.513 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:37.058+0000 7f3e57873700 1 -- 172.21.15.46:0/1348273396 shutdown_connections 2023-06-23T01:01:37.513 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:37.058+0000 7f3e57873700 1 -- 172.21.15.46:0/1348273396 wait complete. 2023-06-23T01:01:39.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:39 smithi046 ceph-mon[84949]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:01:39.647 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwfVdaEdjxWRLztCaJ41kP7iCLhF9/XDXS4eLspHS4XaQr02n4Ko2a6LeYOwLTSrUD6jt6db+IXUI7Sv8zm4j9CGyJbXuMekJbqNB0GZscdv/9jm7E3ly3KYn2FV//DBn8wC+DUliZLciUmVg4VTAaIW4UO0OLEYlm+ZDz8DRiA+LZ6/gbEQBF6cL4xz/hsAx2dTn2kpkPM13q/O6L2cfzhVkt+IMJspFbmSuxIkWRL4W+UFbZbXo4bjVagNDs3pNudF5FkjAqYLW8qrL+OkOtrsPDakeqHBRrDRviRpYZlQnRMV72nKO5ROYxSuyyGmMISQ8QZkdIcw/UnpPcnQWmiuwAcY4OfL8xrEQUq+0ttmjWrw0IRirYbnaebUZrQ/q3A6EA3t2a3S2RsGvXIRUEzWiPmfE+At6WhJB33vTliOeMDmNw9a/6e9JTMjvmqOrI62FTQGABc5pyeeS9Gclx4QHD4pZPYEchnOWLJPi7uPiLSiD2oISDxENwwdb4wAk= ceph-458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:01:39.648 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.965+0000 7f67a2964700 1 Processor -- start 2023-06-23T01:01:39.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.965+0000 7f67a2964700 1 -- start start 2023-06-23T01:01:39.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.965+0000 7f67a2964700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c102590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:39.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.965+0000 7f67a2964700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c102ad0 con 0x7f679c102180 2023-06-23T01:01:39.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.965+0000 7f679bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c102590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:39.649 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.965+0000 7f679bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c102590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41300/0 (socket says 172.21.15.46:41300) 2023-06-23T01:01:39.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.965+0000 7f679bfff700 1 -- 172.21.15.46:0/1099330712 learned_addr learned my addr 172.21.15.46:0/1099330712 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:39.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.966+0000 7f679bfff700 1 -- 172.21.15.46:0/1099330712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c102c10 con 0x7f679c102180 2023-06-23T01:01:39.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.966+0000 7f679bfff700 1 --2- 172.21.15.46:0/1099330712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c102590 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f678c0098e0 tx=0x7f678c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5dcbd7f2704970 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:39.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.966+0000 7f679affd700 1 -- 172.21.15.46:0/1099330712 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f678c009e80 con 0x7f679c102180 2023-06-23T01:01:39.650 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.966+0000 7f679affd700 1 -- 172.21.15.46:0/1099330712 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f678c00bce0 con 0x7f679c102180 2023-06-23T01:01:39.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.966+0000 7f679affd700 1 -- 172.21.15.46:0/1099330712 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f678c011c70 con 0x7f679c102180 2023-06-23T01:01:39.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.966+0000 7f67a2964700 1 -- 172.21.15.46:0/1099330712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 msgr2=0x7f679c102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:39.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.966+0000 7f67a2964700 1 --2- 172.21.15.46:0/1099330712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c102590 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f678c0098e0 tx=0x7f678c009250 comp rx=0 tx=0).stop 2023-06-23T01:01:39.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.967+0000 7f67a2964700 1 -- 172.21.15.46:0/1099330712 shutdown_connections 2023-06-23T01:01:39.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.967+0000 7f67a2964700 1 --2- 172.21.15.46:0/1099330712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c102590 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:39.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.967+0000 7f67a2964700 1 -- 172.21.15.46:0/1099330712 >> 172.21.15.46:0/1099330712 conn(0x7f679c0fd920 msgr2=0x7f679c0ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:39.651 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.967+0000 7f67a2964700 1 -- 172.21.15.46:0/1099330712 shutdown_connections 2023-06-23T01:01:39.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.967+0000 7f67a2964700 1 -- 172.21.15.46:0/1099330712 wait complete. 2023-06-23T01:01:39.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.967+0000 7f67a2964700 1 Processor -- start 2023-06-23T01:01:39.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.967+0000 7f67a2964700 1 -- start start 2023-06-23T01:01:39.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f67a2964700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c110e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:39.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f67a2964700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c111370 con 0x7f679c102180 2023-06-23T01:01:39.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f679bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c110e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:39.652 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f679bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c110e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41314/0 (socket says 172.21.15.46:41314) 2023-06-23T01:01:39.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f679bfff700 1 -- 172.21.15.46:0/84197218 learned_addr learned my addr 172.21.15.46:0/84197218 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:39.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f679bfff700 1 -- 172.21.15.46:0/84197218 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c0094f0 con 0x7f679c102180 2023-06-23T01:01:39.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f679bfff700 1 --2- 172.21.15.46:0/84197218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c110e30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f678c011800 tx=0x7f678c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:39.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f6798ff9700 1 -- 172.21.15.46:0/84197218 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f678c006bc0 con 0x7f679c102180 2023-06-23T01:01:39.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f6798ff9700 1 -- 172.21.15.46:0/84197218 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f678c01aac0 con 0x7f679c102180 2023-06-23T01:01:39.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f67a2964700 1 -- 172.21.15.46:0/84197218 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c111570 con 0x7f679c102180 2023-06-23T01:01:39.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f6798ff9700 1 -- 172.21.15.46:0/84197218 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f678c022a50 con 0x7f679c102180 2023-06-23T01:01:39.653 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.968+0000 7f67a2964700 1 -- 172.21.15.46:0/84197218 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c112130 con 0x7f679c102180 2023-06-23T01:01:39.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.969+0000 7f6798ff9700 1 -- 172.21.15.46:0/84197218 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f678c022c80 con 0x7f679c102180 2023-06-23T01:01:39.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.969+0000 7f6798ff9700 1 --2- 172.21.15.46:0/84197218 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f67840365c0 0x7f6784038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:39.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.969+0000 7f6798ff9700 1 -- 172.21.15.46:0/84197218 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f678c04a430 con 0x7f679c102180 2023-06-23T01:01:39.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.969+0000 7f67a2964700 1 -- 172.21.15.46:0/84197218 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c05f060 con 0x7f679c102180 2023-06-23T01:01:39.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.969+0000 7f679b7fe700 1 --2- 172.21.15.46:0/84197218 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f67840365c0 0x7f6784038a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:39.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.970+0000 7f679b7fe700 1 --2- 172.21.15.46:0/84197218 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f67840365c0 0x7f6784038a70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6790009990 tx=0x7f6790006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:39.654 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:38.974+0000 7f6798ff9700 1 -- 172.21.15.46:0/84197218 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f678c018940 con 0x7f679c102180 2023-06-23T01:01:39.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.168+0000 7f67a2964700 1 -- 172.21.15.46:0/84197218 --> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f679c0fecf0 con 0x7f67840365c0 2023-06-23T01:01:39.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.168+0000 7f6798ff9700 1 -- 172.21.15.46:0/84197218 <== mgr.14120 v2:172.21.15.46:6800/2331157358 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f679c0fecf0 con 0x7f67840365c0 2023-06-23T01:01:39.655 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.170+0000 7f67a2964700 1 -- 172.21.15.46:0/84197218 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f67840365c0 msgr2=0x7f6784038a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:39.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.170+0000 7f67a2964700 1 --2- 172.21.15.46:0/84197218 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f67840365c0 0x7f6784038a70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6790009990 tx=0x7f6790006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:39.656 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.170+0000 7f67a2964700 1 -- 172.21.15.46:0/84197218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 msgr2=0x7f679c110e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:39.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.170+0000 7f67a2964700 1 --2- 172.21.15.46:0/84197218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c110e30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f678c011800 tx=0x7f678c009250 comp rx=0 tx=0).stop 2023-06-23T01:01:39.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.170+0000 7f67a2964700 1 -- 172.21.15.46:0/84197218 shutdown_connections 2023-06-23T01:01:39.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.170+0000 7f67a2964700 1 --2- 172.21.15.46:0/84197218 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f67840365c0 0x7f6784038a70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:39.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.170+0000 7f67a2964700 1 --2- 172.21.15.46:0/84197218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f679c102180 0x7f679c110e30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:39.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.170+0000 7f67a2964700 1 -- 172.21.15.46:0/84197218 >> 172.21.15.46:0/84197218 conn(0x7f679c0fd920 msgr2=0x7f679c0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:39.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.170+0000 7f67a2964700 1 -- 172.21.15.46:0/84197218 shutdown_connections 2023-06-23T01:01:39.657 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:39.170+0000 7f67a2964700 1 -- 172.21.15.46:0/84197218 wait complete. 2023-06-23T01:01:39.658 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-06-23T01:01:39.658 INFO:teuthology.orchestra.run.smithi046.stdout:Adding key to root@localhost authorized_keys... 2023-06-23T01:01:39.658 INFO:teuthology.orchestra.run.smithi046.stdout:Adding host smithi046... 2023-06-23T01:01:40.582 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:40 smithi046 ceph-mon[84949]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:01:41.655 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:41 smithi046 ceph-mon[84949]: Deploying cephadm binary to smithi046 2023-06-23T01:01:42.587 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Added host 'smithi046' with addr '172.21.15.46' 2023-06-23T01:01:42.587 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.090+0000 7fbae837f700 1 Processor -- start 2023-06-23T01:01:42.587 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.090+0000 7fbae837f700 1 -- start start 2023-06-23T01:01:42.588 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.091+0000 7fbae837f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0076140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:42.588 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.091+0000 7fbae837f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae0078050 con 0x7fbae0077c40 2023-06-23T01:01:42.588 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.091+0000 7fbae611b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0076140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:42.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.091+0000 7fbae611b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0076140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41320/0 (socket says 172.21.15.46:41320) 2023-06-23T01:01:42.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.091+0000 7fbae611b700 1 -- 172.21.15.46:0/1385387942 learned_addr learned my addr 172.21.15.46:0/1385387942 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:42.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.091+0000 7fbae611b700 1 -- 172.21.15.46:0/1385387942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae0076680 con 0x7fbae0077c40 2023-06-23T01:01:42.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.091+0000 7fbae611b700 1 --2- 172.21.15.46:0/1385387942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0076140 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0098e0 tx=0x7fbadc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62eab7eab42a9d7 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:42.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.092+0000 7fbae5119700 1 -- 172.21.15.46:0/1385387942 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbadc009e80 con 0x7fbae0077c40 2023-06-23T01:01:42.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.092+0000 7fbae5119700 1 -- 172.21.15.46:0/1385387942 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fbadc00bce0 con 0x7fbae0077c40 2023-06-23T01:01:42.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.092+0000 7fbae5119700 1 -- 172.21.15.46:0/1385387942 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbadc011c70 con 0x7fbae0077c40 2023-06-23T01:01:42.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.092+0000 7fbae837f700 1 -- 172.21.15.46:0/1385387942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 msgr2=0x7fbae0076140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:42.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.092+0000 7fbae837f700 1 --2- 172.21.15.46:0/1385387942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0076140 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0098e0 tx=0x7fbadc009250 comp rx=0 tx=0).stop 2023-06-23T01:01:42.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.092+0000 7fbae837f700 1 -- 172.21.15.46:0/1385387942 shutdown_connections 2023-06-23T01:01:42.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.092+0000 7fbae837f700 1 --2- 172.21.15.46:0/1385387942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0076140 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:42.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.092+0000 7fbae837f700 1 -- 172.21.15.46:0/1385387942 >> 172.21.15.46:0/1385387942 conn(0x7fbae00fd900 msgr2=0x7fbae00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:42.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.092+0000 7fbae837f700 1 -- 172.21.15.46:0/1385387942 shutdown_connections 2023-06-23T01:01:42.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.092+0000 7fbae837f700 1 -- 172.21.15.46:0/1385387942 wait complete. 2023-06-23T01:01:42.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.093+0000 7fbae837f700 1 Processor -- start 2023-06-23T01:01:42.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.093+0000 7fbae837f700 1 -- start start 2023-06-23T01:01:42.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.093+0000 7fbae837f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0112f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:42.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.093+0000 7fbae837f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae0113490 con 0x7fbae0077c40 2023-06-23T01:01:42.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.093+0000 7fbae611b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0112f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:42.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.093+0000 7fbae611b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0112f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41334/0 (socket says 172.21.15.46:41334) 2023-06-23T01:01:42.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.093+0000 7fbae611b700 1 -- 172.21.15.46:0/1554715433 learned_addr learned my addr 172.21.15.46:0/1554715433 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:42.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.093+0000 7fbae611b700 1 -- 172.21.15.46:0/1554715433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc0094f0 con 0x7fbae0077c40 2023-06-23T01:01:42.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.094+0000 7fbae611b700 1 --2- 172.21.15.46:0/1554715433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0112f50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbadc011800 tx=0x7fbadc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:42.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.094+0000 7fbad6ffd700 1 -- 172.21.15.46:0/1554715433 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbadc01a640 con 0x7fbae0077c40 2023-06-23T01:01:42.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.094+0000 7fbad6ffd700 1 -- 172.21.15.46:0/1554715433 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fbadc01ac20 con 0x7fbae0077c40 2023-06-23T01:01:42.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.094+0000 7fbad6ffd700 1 -- 172.21.15.46:0/1554715433 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbadc0229e0 con 0x7fbae0077c40 2023-06-23T01:01:42.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.094+0000 7fbae837f700 1 -- 172.21.15.46:0/1554715433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae0113690 con 0x7fbae0077c40 2023-06-23T01:01:42.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.094+0000 7fbae837f700 1 -- 172.21.15.46:0/1554715433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae0114330 con 0x7fbae0077c40 2023-06-23T01:01:42.599 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.095+0000 7fbad6ffd700 1 -- 172.21.15.46:0/1554715433 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7fbadc01a7a0 con 0x7fbae0077c40 2023-06-23T01:01:42.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.095+0000 7fbad6ffd700 1 --2- 172.21.15.46:0/1554715433 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fbacc0365c0 0x7fbacc038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:42.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.095+0000 7fbad6ffd700 1 -- 172.21.15.46:0/1554715433 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbadc04a470 con 0x7fbae0077c40 2023-06-23T01:01:42.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.095+0000 7fbae837f700 1 -- 172.21.15.46:0/1554715433 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae005f060 con 0x7fbae0077c40 2023-06-23T01:01:42.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.095+0000 7fbae591a700 1 --2- 172.21.15.46:0/1554715433 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fbacc0365c0 0x7fbacc038a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:42.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.095+0000 7fbae591a700 1 --2- 172.21.15.46:0/1554715433 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fbacc0365c0 0x7fbacc038a70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009990 tx=0x7fbad0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:42.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.100+0000 7fbad6ffd700 1 -- 172.21.15.46:0/1554715433 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fbadc022400 con 0x7fbae0077c40 2023-06-23T01:01:42.600 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:40.298+0000 7fbae837f700 1 -- 172.21.15.46:0/1554715433 --> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}) v1 -- 0x7fbae00fec80 con 0x7fbacc0365c0 2023-06-23T01:01:42.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:42.135+0000 7fbad6ffd700 1 -- 172.21.15.46:0/1554715433 <== mgr.14120 v2:172.21.15.46:6800/2331157358 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fbae00fec80 con 0x7fbacc0365c0 2023-06-23T01:01:42.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:42.138+0000 7fbae837f700 1 -- 172.21.15.46:0/1554715433 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fbacc0365c0 msgr2=0x7fbacc038a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:42.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:42.138+0000 7fbae837f700 1 --2- 172.21.15.46:0/1554715433 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fbacc0365c0 0x7fbacc038a70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009990 tx=0x7fbad0006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:42.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:42.138+0000 7fbae837f700 1 -- 172.21.15.46:0/1554715433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 msgr2=0x7fbae0112f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:42.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:42.138+0000 7fbae837f700 1 --2- 172.21.15.46:0/1554715433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0112f50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbadc011800 tx=0x7fbadc009250 comp rx=0 tx=0).stop 2023-06-23T01:01:42.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:42.138+0000 7fbae837f700 1 -- 172.21.15.46:0/1554715433 shutdown_connections 2023-06-23T01:01:42.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:42.138+0000 7fbae837f700 1 --2- 172.21.15.46:0/1554715433 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fbacc0365c0 0x7fbacc038a70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:42.601 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:42.138+0000 7fbae837f700 1 --2- 172.21.15.46:0/1554715433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbae0077c40 0x7fbae0112f50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:42.602 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:42.138+0000 7fbae837f700 1 -- 172.21.15.46:0/1554715433 >> 172.21.15.46:0/1554715433 conn(0x7fbae00fd900 msgr2=0x7fbae00fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:42.602 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:42.138+0000 7fbae837f700 1 -- 172.21.15.46:0/1554715433 shutdown_connections 2023-06-23T01:01:42.602 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:42.138+0000 7fbae837f700 1 -- 172.21.15.46:0/1554715433 wait complete. 2023-06-23T01:01:42.602 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying unmanaged mon service... 2023-06-23T01:01:43.539 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:43 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:43.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:43 smithi046 ceph-mon[84949]: Added host smithi046 2023-06-23T01:01:43.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:43 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:01:44.037 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-06-23T01:01:44.037 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.094+0000 7fe2976ff700 1 Processor -- start 2023-06-23T01:01:44.037 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.094+0000 7fe2976ff700 1 -- start start 2023-06-23T01:01:44.037 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.094+0000 7fe2976ff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe290101430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:44.037 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.094+0000 7fe2976ff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe290101970 con 0x7fe290101020 2023-06-23T01:01:44.037 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.095+0000 7fe29549b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe290101430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:44.038 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.095+0000 7fe29549b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe290101430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41350/0 (socket says 172.21.15.46:41350) 2023-06-23T01:01:44.038 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.095+0000 7fe29549b700 1 -- 172.21.15.46:0/3590556095 learned_addr learned my addr 172.21.15.46:0/3590556095 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:44.038 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.095+0000 7fe29549b700 1 -- 172.21.15.46:0/3590556095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe290101ab0 con 0x7fe290101020 2023-06-23T01:01:44.038 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.095+0000 7fe29549b700 1 --2- 172.21.15.46:0/3590556095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe290101430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe288009e50 tx=0x7fe288009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead4c571c28dea52 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:44.038 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.095+0000 7fe28ffff700 1 -- 172.21.15.46:0/3590556095 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe28800e070 con 0x7fe290101020 2023-06-23T01:01:44.038 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.095+0000 7fe28ffff700 1 -- 172.21.15.46:0/3590556095 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe288012430 con 0x7fe290101020 2023-06-23T01:01:44.038 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.095+0000 7fe28ffff700 1 -- 172.21.15.46:0/3590556095 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe28800bea0 con 0x7fe290101020 2023-06-23T01:01:44.039 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.096+0000 7fe2976ff700 1 -- 172.21.15.46:0/3590556095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 msgr2=0x7fe290101430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:44.039 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.096+0000 7fe2976ff700 1 --2- 172.21.15.46:0/3590556095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe290101430 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe288009e50 tx=0x7fe288009840 comp rx=0 tx=0).stop 2023-06-23T01:01:44.039 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.096+0000 7fe2976ff700 1 -- 172.21.15.46:0/3590556095 shutdown_connections 2023-06-23T01:01:44.039 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.096+0000 7fe2976ff700 1 --2- 172.21.15.46:0/3590556095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe290101430 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:44.039 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.096+0000 7fe2976ff700 1 -- 172.21.15.46:0/3590556095 >> 172.21.15.46:0/3590556095 conn(0x7fe2900fc7e0 msgr2=0x7fe2900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:44.039 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.096+0000 7fe2976ff700 1 -- 172.21.15.46:0/3590556095 shutdown_connections 2023-06-23T01:01:44.039 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.096+0000 7fe2976ff700 1 -- 172.21.15.46:0/3590556095 wait complete. 2023-06-23T01:01:44.040 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.096+0000 7fe2976ff700 1 Processor -- start 2023-06-23T01:01:44.040 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.096+0000 7fe2976ff700 1 -- start start 2023-06-23T01:01:44.040 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.097+0000 7fe2976ff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe29010fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:44.040 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.097+0000 7fe2976ff700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2901102c0 con 0x7fe290101020 2023-06-23T01:01:44.040 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.097+0000 7fe29549b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe29010fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:44.040 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.097+0000 7fe29549b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe29010fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41354/0 (socket says 172.21.15.46:41354) 2023-06-23T01:01:44.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.097+0000 7fe29549b700 1 -- 172.21.15.46:0/1621348807 learned_addr learned my addr 172.21.15.46:0/1621348807 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:44.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.097+0000 7fe29549b700 1 -- 172.21.15.46:0/1621348807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe288009ae0 con 0x7fe290101020 2023-06-23T01:01:44.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.097+0000 7fe29549b700 1 --2- 172.21.15.46:0/1621348807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe29010fd80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe288013fd0 tx=0x7fe288009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:44.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.097+0000 7fe28dffb700 1 -- 172.21.15.46:0/1621348807 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe28800e040 con 0x7fe290101020 2023-06-23T01:01:44.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.097+0000 7fe2976ff700 1 -- 172.21.15.46:0/1621348807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2901104c0 con 0x7fe290101020 2023-06-23T01:01:44.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.097+0000 7fe28dffb700 1 -- 172.21.15.46:0/1621348807 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe2880092e0 con 0x7fe290101020 2023-06-23T01:01:44.041 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.097+0000 7fe28dffb700 1 -- 172.21.15.46:0/1621348807 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe28801c6b0 con 0x7fe290101020 2023-06-23T01:01:44.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.098+0000 7fe2976ff700 1 -- 172.21.15.46:0/1621348807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe290111080 con 0x7fe290101020 2023-06-23T01:01:44.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.098+0000 7fe28dffb700 1 -- 172.21.15.46:0/1621348807 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7fe28801e660 con 0x7fe290101020 2023-06-23T01:01:44.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.099+0000 7fe2976ff700 1 -- 172.21.15.46:0/1621348807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29005f060 con 0x7fe290101020 2023-06-23T01:01:44.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.099+0000 7fe28dffb700 1 --2- 172.21.15.46:0/1621348807 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fe278036560 0x7fe278038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:44.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.099+0000 7fe28dffb700 1 -- 172.21.15.46:0/1621348807 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe28804a1e0 con 0x7fe290101020 2023-06-23T01:01:44.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.099+0000 7fe294c9a700 1 --2- 172.21.15.46:0/1621348807 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fe278036560 0x7fe278038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:44.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.099+0000 7fe294c9a700 1 --2- 172.21.15.46:0/1621348807 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fe278036560 0x7fe278038a10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe284009990 tx=0x7fe284006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:44.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.104+0000 7fe28dffb700 1 -- 172.21.15.46:0/1621348807 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fe288007500 con 0x7fe290101020 2023-06-23T01:01:44.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.315+0000 7fe2976ff700 1 -- 172.21.15.46:0/1621348807 --> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe2900fdbe0 con 0x7fe278036560 2023-06-23T01:01:44.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.319+0000 7fe28dffb700 1 -- 172.21.15.46:0/1621348807 <== mgr.14120 v2:172.21.15.46:6800/2331157358 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe2900fdbe0 con 0x7fe278036560 2023-06-23T01:01:44.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.321+0000 7fe2976ff700 1 -- 172.21.15.46:0/1621348807 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fe278036560 msgr2=0x7fe278038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:44.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.321+0000 7fe2976ff700 1 --2- 172.21.15.46:0/1621348807 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fe278036560 0x7fe278038a10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe284009990 tx=0x7fe284006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:44.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.321+0000 7fe2976ff700 1 -- 172.21.15.46:0/1621348807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 msgr2=0x7fe29010fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:44.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.321+0000 7fe2976ff700 1 --2- 172.21.15.46:0/1621348807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe29010fd80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe288013fd0 tx=0x7fe288009840 comp rx=0 tx=0).stop 2023-06-23T01:01:44.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.321+0000 7fe2976ff700 1 -- 172.21.15.46:0/1621348807 shutdown_connections 2023-06-23T01:01:44.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.321+0000 7fe2976ff700 1 --2- 172.21.15.46:0/1621348807 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fe278036560 0x7fe278038a10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:44.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.321+0000 7fe2976ff700 1 --2- 172.21.15.46:0/1621348807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe290101020 0x7fe29010fd80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:44.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.321+0000 7fe2976ff700 1 -- 172.21.15.46:0/1621348807 >> 172.21.15.46:0/1621348807 conn(0x7fe2900fc7e0 msgr2=0x7fe2900fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:44.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.322+0000 7fe2976ff700 1 -- 172.21.15.46:0/1621348807 shutdown_connections 2023-06-23T01:01:44.044 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:43.322+0000 7fe2976ff700 1 -- 172.21.15.46:0/1621348807 wait complete. 2023-06-23T01:01:44.044 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying unmanaged mgr service... 2023-06-23T01:01:44.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:44 smithi046 ceph-mon[84949]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:01:44.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:44 smithi046 ceph-mon[84949]: Saving service mon spec with placement count:5 2023-06-23T01:01:44.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:44 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:45.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-06-23T01:01:45.176 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.539+0000 7fa69938c700 1 Processor -- start 2023-06-23T01:01:45.176 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.539+0000 7fa69938c700 1 -- start start 2023-06-23T01:01:45.176 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.539+0000 7fa69938c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694107130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:45.176 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.539+0000 7fa69938c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa694102b20 con 0x7fa694070e00 2023-06-23T01:01:45.176 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.539+0000 7fa692ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694107130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:45.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.539+0000 7fa692ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694107130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47630/0 (socket says 172.21.15.46:47630) 2023-06-23T01:01:45.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.539+0000 7fa692ffd700 1 -- 172.21.15.46:0/2577941855 learned_addr learned my addr 172.21.15.46:0/2577941855 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:45.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.540+0000 7fa692ffd700 1 -- 172.21.15.46:0/2577941855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa694107670 con 0x7fa694070e00 2023-06-23T01:01:45.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.540+0000 7fa692ffd700 1 --2- 172.21.15.46:0/2577941855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694107130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa6840098e0 tx=0x7fa684009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a037b2bd5093132a server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:45.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.540+0000 7fa691ffb700 1 -- 172.21.15.46:0/2577941855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa684009e80 con 0x7fa694070e00 2023-06-23T01:01:45.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.540+0000 7fa691ffb700 1 -- 172.21.15.46:0/2577941855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fa68400bce0 con 0x7fa694070e00 2023-06-23T01:01:45.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.540+0000 7fa691ffb700 1 -- 172.21.15.46:0/2577941855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa684011c70 con 0x7fa694070e00 2023-06-23T01:01:45.177 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.541+0000 7fa69938c700 1 -- 172.21.15.46:0/2577941855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 msgr2=0x7fa694107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:45.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.541+0000 7fa69938c700 1 --2- 172.21.15.46:0/2577941855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694107130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa6840098e0 tx=0x7fa684009250 comp rx=0 tx=0).stop 2023-06-23T01:01:45.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.541+0000 7fa69938c700 1 -- 172.21.15.46:0/2577941855 shutdown_connections 2023-06-23T01:01:45.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.541+0000 7fa69938c700 1 --2- 172.21.15.46:0/2577941855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694107130 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:45.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.541+0000 7fa69938c700 1 -- 172.21.15.46:0/2577941855 >> 172.21.15.46:0/2577941855 conn(0x7fa6940fc7e0 msgr2=0x7fa6940febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:45.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.541+0000 7fa69938c700 1 -- 172.21.15.46:0/2577941855 shutdown_connections 2023-06-23T01:01:45.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.541+0000 7fa69938c700 1 -- 172.21.15.46:0/2577941855 wait complete. 2023-06-23T01:01:45.178 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.541+0000 7fa69938c700 1 Processor -- start 2023-06-23T01:01:45.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.541+0000 7fa69938c700 1 -- start start 2023-06-23T01:01:45.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa69938c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694112f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:45.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa69938c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa694113480 con 0x7fa694070e00 2023-06-23T01:01:45.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa692ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694112f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:45.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa692ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694112f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47636/0 (socket says 172.21.15.46:47636) 2023-06-23T01:01:45.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa692ffd700 1 -- 172.21.15.46:0/3560741298 learned_addr learned my addr 172.21.15.46:0/3560741298 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:45.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa692ffd700 1 -- 172.21.15.46:0/3560741298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6840094f0 con 0x7fa694070e00 2023-06-23T01:01:45.179 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa692ffd700 1 --2- 172.21.15.46:0/3560741298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694112f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa684011800 tx=0x7fa684009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:45.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa67b7fe700 1 -- 172.21.15.46:0/3560741298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa684006bc0 con 0x7fa694070e00 2023-06-23T01:01:45.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa67b7fe700 1 -- 172.21.15.46:0/3560741298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fa68401aac0 con 0x7fa694070e00 2023-06-23T01:01:45.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa69938c700 1 -- 172.21.15.46:0/3560741298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa694113680 con 0x7fa694070e00 2023-06-23T01:01:45.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa67b7fe700 1 -- 172.21.15.46:0/3560741298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa684022a20 con 0x7fa694070e00 2023-06-23T01:01:45.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.542+0000 7fa69938c700 1 -- 172.21.15.46:0/3560741298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa694114320 con 0x7fa694070e00 2023-06-23T01:01:45.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.543+0000 7fa69938c700 1 -- 172.21.15.46:0/3560741298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69405f060 con 0x7fa694070e00 2023-06-23T01:01:45.180 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.544+0000 7fa67b7fe700 1 -- 172.21.15.46:0/3560741298 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7fa684022400 con 0x7fa694070e00 2023-06-23T01:01:45.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.544+0000 7fa67b7fe700 1 --2- 172.21.15.46:0/3560741298 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa67c0365c0 0x7fa67c038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:45.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.544+0000 7fa67b7fe700 1 -- 172.21.15.46:0/3560741298 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa68404a450 con 0x7fa694070e00 2023-06-23T01:01:45.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.544+0000 7fa6927fc700 1 --2- 172.21.15.46:0/3560741298 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa67c0365c0 0x7fa67c038a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:45.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.544+0000 7fa6927fc700 1 --2- 172.21.15.46:0/3560741298 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa67c0365c0 0x7fa67c038a70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa688009990 tx=0x7fa688006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:45.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.547+0000 7fa67b7fe700 1 -- 172.21.15.46:0/3560741298 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fa684018350 con 0x7fa694070e00 2023-06-23T01:01:45.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.755+0000 7fa69938c700 1 -- 172.21.15.46:0/3560741298 --> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa6940fdb50 con 0x7fa67c0365c0 2023-06-23T01:01:45.181 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.760+0000 7fa67b7fe700 1 -- 172.21.15.46:0/3560741298 <== mgr.14120 v2:172.21.15.46:6800/2331157358 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa6940fdb50 con 0x7fa67c0365c0 2023-06-23T01:01:45.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.762+0000 7fa69938c700 1 -- 172.21.15.46:0/3560741298 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa67c0365c0 msgr2=0x7fa67c038a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:45.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.762+0000 7fa69938c700 1 --2- 172.21.15.46:0/3560741298 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa67c0365c0 0x7fa67c038a70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa688009990 tx=0x7fa688006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:45.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.762+0000 7fa69938c700 1 -- 172.21.15.46:0/3560741298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 msgr2=0x7fa694112f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:45.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.762+0000 7fa69938c700 1 --2- 172.21.15.46:0/3560741298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694112f40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa684011800 tx=0x7fa684009250 comp rx=0 tx=0).stop 2023-06-23T01:01:45.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.763+0000 7fa69938c700 1 -- 172.21.15.46:0/3560741298 shutdown_connections 2023-06-23T01:01:45.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.763+0000 7fa69938c700 1 --2- 172.21.15.46:0/3560741298 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa67c0365c0 0x7fa67c038a70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:45.182 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.763+0000 7fa69938c700 1 --2- 172.21.15.46:0/3560741298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa694070e00 0x7fa694112f40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:45.183 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.763+0000 7fa69938c700 1 -- 172.21.15.46:0/3560741298 >> 172.21.15.46:0/3560741298 conn(0x7fa6940fc7e0 msgr2=0x7fa6940fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:45.183 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.763+0000 7fa69938c700 1 -- 172.21.15.46:0/3560741298 shutdown_connections 2023-06-23T01:01:45.183 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:44.763+0000 7fa69938c700 1 -- 172.21.15.46:0/3560741298 wait complete. 2023-06-23T01:01:46.055 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:45 smithi046 ceph-mon[84949]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:01:46.055 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:45 smithi046 ceph-mon[84949]: Saving service mgr spec with placement count:2 2023-06-23T01:01:46.056 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:45 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:46.056 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:45 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:46.056 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:45 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:46.212 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.619+0000 7fa9d7008700 1 Processor -- start 2023-06-23T01:01:46.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.619+0000 7fa9d7008700 1 -- start start 2023-06-23T01:01:46.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.619+0000 7fa9d7008700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0101490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:46.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.619+0000 7fa9d7008700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d01019d0 con 0x7fa9d0101080 2023-06-23T01:01:46.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.619+0000 7fa9d4da4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:46.213 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.619+0000 7fa9d4da4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0101490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47638/0 (socket says 172.21.15.46:47638) 2023-06-23T01:01:46.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.619+0000 7fa9d4da4700 1 -- 172.21.15.46:0/3695427614 learned_addr learned my addr 172.21.15.46:0/3695427614 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:46.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.620+0000 7fa9d4da4700 1 -- 172.21.15.46:0/3695427614 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d0101b10 con 0x7fa9d0101080 2023-06-23T01:01:46.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.620+0000 7fa9d4da4700 1 --2- 172.21.15.46:0/3695427614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0101490 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40098e0 tx=0x7fa9c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de95e4810b5c163 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:46.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.620+0000 7fa9cf7fe700 1 -- 172.21.15.46:0/3695427614 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9c4009e80 con 0x7fa9d0101080 2023-06-23T01:01:46.214 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.620+0000 7fa9cf7fe700 1 -- 172.21.15.46:0/3695427614 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa9c400bd00 con 0x7fa9d0101080 2023-06-23T01:01:46.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.620+0000 7fa9cf7fe700 1 -- 172.21.15.46:0/3695427614 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9c4011c00 con 0x7fa9d0101080 2023-06-23T01:01:46.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.620+0000 7fa9d7008700 1 -- 172.21.15.46:0/3695427614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 msgr2=0x7fa9d0101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:46.217 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.621+0000 7fa9d7008700 1 --2- 172.21.15.46:0/3695427614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0101490 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40098e0 tx=0x7fa9c4009250 comp rx=0 tx=0).stop 2023-06-23T01:01:46.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.621+0000 7fa9d7008700 1 -- 172.21.15.46:0/3695427614 shutdown_connections 2023-06-23T01:01:46.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.621+0000 7fa9d7008700 1 --2- 172.21.15.46:0/3695427614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0101490 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:46.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.621+0000 7fa9d7008700 1 -- 172.21.15.46:0/3695427614 >> 172.21.15.46:0/3695427614 conn(0x7fa9d00fc820 msgr2=0x7fa9d00fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:46.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.621+0000 7fa9d7008700 1 -- 172.21.15.46:0/3695427614 shutdown_connections 2023-06-23T01:01:46.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.621+0000 7fa9d7008700 1 -- 172.21.15.46:0/3695427614 wait complete. 2023-06-23T01:01:46.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.621+0000 7fa9d7008700 1 Processor -- start 2023-06-23T01:01:46.218 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.621+0000 7fa9d7008700 1 -- start start 2023-06-23T01:01:46.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.622+0000 7fa9d7008700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:46.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.622+0000 7fa9d7008700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d0070d80 con 0x7fa9d0101080 2023-06-23T01:01:46.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.622+0000 7fa9d4da4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:46.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.622+0000 7fa9d4da4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47648/0 (socket says 172.21.15.46:47648) 2023-06-23T01:01:46.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.622+0000 7fa9d4da4700 1 -- 172.21.15.46:0/3416245013 learned_addr learned my addr 172.21.15.46:0/3416245013 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:46.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.622+0000 7fa9d4da4700 1 -- 172.21.15.46:0/3416245013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c40094f0 con 0x7fa9d0101080 2023-06-23T01:01:46.219 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.622+0000 7fa9d4da4700 1 --2- 172.21.15.46:0/3416245013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0113ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4000c00 tx=0x7fa9c4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:46.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.622+0000 7fa9cd7fa700 1 -- 172.21.15.46:0/3416245013 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9c401a640 con 0x7fa9d0101080 2023-06-23T01:01:46.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.622+0000 7fa9cd7fa700 1 -- 172.21.15.46:0/3416245013 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa9c401ac40 con 0x7fa9d0101080 2023-06-23T01:01:46.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.622+0000 7fa9d7008700 1 -- 172.21.15.46:0/3416245013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d0070f80 con 0x7fa9d0101080 2023-06-23T01:01:46.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.622+0000 7fa9cd7fa700 1 -- 172.21.15.46:0/3416245013 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9c4022b10 con 0x7fa9d0101080 2023-06-23T01:01:46.220 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.623+0000 7fa9d7008700 1 -- 172.21.15.46:0/3416245013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d01141e0 con 0x7fa9d0101080 2023-06-23T01:01:46.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.623+0000 7fa9cd7fa700 1 -- 172.21.15.46:0/3416245013 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7fa9c4022d40 con 0x7fa9d0101080 2023-06-23T01:01:46.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.623+0000 7fa9d7008700 1 -- 172.21.15.46:0/3416245013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d004b970 con 0x7fa9d0101080 2023-06-23T01:01:46.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.624+0000 7fa9cd7fa700 1 --2- 172.21.15.46:0/3416245013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa9c00365c0 0x7fa9c0038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:46.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.624+0000 7fa9cd7fa700 1 -- 172.21.15.46:0/3416245013 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa9c404a7b0 con 0x7fa9d0101080 2023-06-23T01:01:46.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.624+0000 7fa9cffff700 1 --2- 172.21.15.46:0/3416245013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa9c00365c0 0x7fa9c0038a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:46.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.624+0000 7fa9cffff700 1 --2- 172.21.15.46:0/3416245013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa9c00365c0 0x7fa9c0038a70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc009990 tx=0x7fa9bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:46.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.629+0000 7fa9cd7fa700 1 -- 172.21.15.46:0/3416245013 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fa9c4018710 con 0x7fa9d0101080 2023-06-23T01:01:46.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.830+0000 7fa9d7008700 1 -- 172.21.15.46:0/3416245013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fa9d0102f20 con 0x7fa9d0101080 2023-06-23T01:01:46.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.835+0000 7fa9cd7fa700 1 -- 172.21.15.46:0/3416245013 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fa9c40213a0 con 0x7fa9d0101080 2023-06-23T01:01:46.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.837+0000 7fa9d7008700 1 -- 172.21.15.46:0/3416245013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa9c00365c0 msgr2=0x7fa9c0038a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:46.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.837+0000 7fa9d7008700 1 --2- 172.21.15.46:0/3416245013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa9c00365c0 0x7fa9c0038a70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc009990 tx=0x7fa9bc006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:46.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.837+0000 7fa9d7008700 1 -- 172.21.15.46:0/3416245013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 msgr2=0x7fa9d0113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:46.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.837+0000 7fa9d7008700 1 --2- 172.21.15.46:0/3416245013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0113ae0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4000c00 tx=0x7fa9c4009250 comp rx=0 tx=0).stop 2023-06-23T01:01:46.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.837+0000 7fa9d7008700 1 -- 172.21.15.46:0/3416245013 shutdown_connections 2023-06-23T01:01:46.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.837+0000 7fa9d7008700 1 --2- 172.21.15.46:0/3416245013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa9c00365c0 0x7fa9c0038a70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:46.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.837+0000 7fa9d7008700 1 --2- 172.21.15.46:0/3416245013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9d0101080 0x7fa9d0113ae0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:46.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.837+0000 7fa9d7008700 1 -- 172.21.15.46:0/3416245013 >> 172.21.15.46:0/3416245013 conn(0x7fa9d00fc820 msgr2=0x7fa9d00fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:46.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.838+0000 7fa9d7008700 1 -- 172.21.15.46:0/3416245013 shutdown_connections 2023-06-23T01:01:46.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:45.838+0000 7fa9d7008700 1 -- 172.21.15.46:0/3416245013 wait complete. 2023-06-23T01:01:47.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:46 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3416245013' entity='client.admin' 2023-06-23T01:01:47.359 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.726+0000 7f3e9749f700 1 Processor -- start 2023-06-23T01:01:47.360 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.726+0000 7f3e9749f700 1 -- start start 2023-06-23T01:01:47.360 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.726+0000 7f3e9749f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90103bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:47.360 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.726+0000 7f3e9749f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e901040f0 con 0x7f3e901037a0 2023-06-23T01:01:47.360 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.727+0000 7f3e9523b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90103bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:47.360 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.727+0000 7f3e9523b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90103bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47652/0 (socket says 172.21.15.46:47652) 2023-06-23T01:01:47.361 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.727+0000 7f3e9523b700 1 -- 172.21.15.46:0/2379868755 learned_addr learned my addr 172.21.15.46:0/2379868755 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:47.361 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.727+0000 7f3e9523b700 1 -- 172.21.15.46:0/2379868755 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e90104230 con 0x7f3e901037a0 2023-06-23T01:01:47.361 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.727+0000 7f3e9523b700 1 --2- 172.21.15.46:0/2379868755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90103bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0098e0 tx=0x7f3e8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41d63819a6e57d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:47.361 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.727+0000 7f3e87fff700 1 -- 172.21.15.46:0/2379868755 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e8c009e80 con 0x7f3e901037a0 2023-06-23T01:01:47.361 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.728+0000 7f3e87fff700 1 -- 172.21.15.46:0/2379868755 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3e8c00bd00 con 0x7f3e901037a0 2023-06-23T01:01:47.361 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.728+0000 7f3e87fff700 1 -- 172.21.15.46:0/2379868755 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e8c011c00 con 0x7f3e901037a0 2023-06-23T01:01:47.362 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.728+0000 7f3e9749f700 1 -- 172.21.15.46:0/2379868755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 msgr2=0x7f3e90103bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:47.362 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.728+0000 7f3e9749f700 1 --2- 172.21.15.46:0/2379868755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90103bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0098e0 tx=0x7f3e8c009250 comp rx=0 tx=0).stop 2023-06-23T01:01:47.362 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.728+0000 7f3e9749f700 1 -- 172.21.15.46:0/2379868755 shutdown_connections 2023-06-23T01:01:47.362 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.728+0000 7f3e9749f700 1 --2- 172.21.15.46:0/2379868755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90103bb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:47.362 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.728+0000 7f3e9749f700 1 -- 172.21.15.46:0/2379868755 >> 172.21.15.46:0/2379868755 conn(0x7f3e900fbf90 msgr2=0x7f3e900fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:47.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.728+0000 7f3e9749f700 1 -- 172.21.15.46:0/2379868755 shutdown_connections 2023-06-23T01:01:47.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.728+0000 7f3e9749f700 1 -- 172.21.15.46:0/2379868755 wait complete. 2023-06-23T01:01:47.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.729+0000 7f3e9749f700 1 Processor -- start 2023-06-23T01:01:47.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.729+0000 7f3e9749f700 1 -- start start 2023-06-23T01:01:47.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.729+0000 7f3e9749f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90110e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:47.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.729+0000 7f3e9749f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e90111370 con 0x7f3e901037a0 2023-06-23T01:01:47.363 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.729+0000 7f3e9523b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90110e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:47.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.729+0000 7f3e9523b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90110e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47664/0 (socket says 172.21.15.46:47664) 2023-06-23T01:01:47.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.729+0000 7f3e9523b700 1 -- 172.21.15.46:0/4015871008 learned_addr learned my addr 172.21.15.46:0/4015871008 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:47.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.729+0000 7f3e9523b700 1 -- 172.21.15.46:0/4015871008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c0094f0 con 0x7f3e901037a0 2023-06-23T01:01:47.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.730+0000 7f3e9523b700 1 --2- 172.21.15.46:0/4015871008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90110e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c006fd0 tx=0x7f3e8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:47.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.730+0000 7f3e85ffb700 1 -- 172.21.15.46:0/4015871008 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e8c01a670 con 0x7f3e901037a0 2023-06-23T01:01:47.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.730+0000 7f3e85ffb700 1 -- 172.21.15.46:0/4015871008 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3e8c01ac70 con 0x7f3e901037a0 2023-06-23T01:01:47.364 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.730+0000 7f3e9749f700 1 -- 172.21.15.46:0/4015871008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e90111570 con 0x7f3e901037a0 2023-06-23T01:01:47.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.730+0000 7f3e85ffb700 1 -- 172.21.15.46:0/4015871008 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3e8c022a70 con 0x7f3e901037a0 2023-06-23T01:01:47.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.730+0000 7f3e9749f700 1 -- 172.21.15.46:0/4015871008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e90112130 con 0x7f3e901037a0 2023-06-23T01:01:47.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.731+0000 7f3e85ffb700 1 -- 172.21.15.46:0/4015871008 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7f3e8c022cb0 con 0x7f3e901037a0 2023-06-23T01:01:47.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.731+0000 7f3e9749f700 1 -- 172.21.15.46:0/4015871008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9005f060 con 0x7f3e901037a0 2023-06-23T01:01:47.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.731+0000 7f3e85ffb700 1 --2- 172.21.15.46:0/4015871008 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e7c036560 0x7f3e7c038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:47.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.731+0000 7f3e85ffb700 1 -- 172.21.15.46:0/4015871008 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3e8c049cf0 con 0x7f3e901037a0 2023-06-23T01:01:47.365 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.731+0000 7f3e94a3a700 1 --2- 172.21.15.46:0/4015871008 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e7c036560 0x7f3e7c038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:47.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.732+0000 7f3e94a3a700 1 --2- 172.21.15.46:0/4015871008 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e7c036560 0x7f3e7c038a10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009990 tx=0x7f3e80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:47.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.736+0000 7f3e85ffb700 1 -- 172.21.15.46:0/4015871008 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f3e8c04e070 con 0x7f3e901037a0 2023-06-23T01:01:47.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.932+0000 7f3e9749f700 1 -- 172.21.15.46:0/4015871008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f3e9004b970 con 0x7f3e901037a0 2023-06-23T01:01:47.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.935+0000 7f3e85ffb700 1 -- 172.21.15.46:0/4015871008 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f3e8c01e030 con 0x7f3e901037a0 2023-06-23T01:01:47.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.937+0000 7f3e9749f700 1 -- 172.21.15.46:0/4015871008 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e7c036560 msgr2=0x7f3e7c038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:47.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.937+0000 7f3e9749f700 1 --2- 172.21.15.46:0/4015871008 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e7c036560 0x7f3e7c038a10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009990 tx=0x7f3e80006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:47.366 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.937+0000 7f3e9749f700 1 -- 172.21.15.46:0/4015871008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 msgr2=0x7f3e90110e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:47.367 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.937+0000 7f3e9749f700 1 --2- 172.21.15.46:0/4015871008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90110e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c006fd0 tx=0x7f3e8c009250 comp rx=0 tx=0).stop 2023-06-23T01:01:47.367 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.937+0000 7f3e9749f700 1 -- 172.21.15.46:0/4015871008 shutdown_connections 2023-06-23T01:01:47.367 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.937+0000 7f3e9749f700 1 --2- 172.21.15.46:0/4015871008 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f3e7c036560 0x7f3e7c038a10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:47.367 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.937+0000 7f3e9749f700 1 --2- 172.21.15.46:0/4015871008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e901037a0 0x7f3e90110e30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:47.367 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.937+0000 7f3e9749f700 1 -- 172.21.15.46:0/4015871008 >> 172.21.15.46:0/4015871008 conn(0x7f3e900fbf90 msgr2=0x7f3e900fcc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:47.367 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.938+0000 7f3e9749f700 1 -- 172.21.15.46:0/4015871008 shutdown_connections 2023-06-23T01:01:47.368 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:46.938+0000 7f3e9749f700 1 -- 172.21.15.46:0/4015871008 wait complete. 2023-06-23T01:01:47.368 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling the dashboard module... 2023-06-23T01:01:48.317 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:47 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/4015871008' entity='client.admin' 2023-06-23T01:01:48.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:47 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:49.205 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:48 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3135666210' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-06-23T01:01:49.205 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:48 smithi046 ceph-mon[84949]: from='mgr.14120 172.21.15.46:0/2933215491' entity='mgr.x' 2023-06-23T01:01:49.206 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:48 smithi046 conmon[85362]: ignoring --setuser ceph since I am not root 2023-06-23T01:01:49.206 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:48 smithi046 conmon[85362]: ignoring --setgroup ceph since I am not root 2023-06-23T01:01:49.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.827+0000 7fc604bb6700 1 Processor -- start 2023-06-23T01:01:49.399 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.827+0000 7fc604bb6700 1 -- start start 2023-06-23T01:01:49.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.827+0000 7fc604bb6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc600101430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:49.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.827+0000 7fc604bb6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc600101970 con 0x7fc600101020 2023-06-23T01:01:49.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.827+0000 7fc5fe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc600101430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:49.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.827+0000 7fc5fe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc600101430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47666/0 (socket says 172.21.15.46:47666) 2023-06-23T01:01:49.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.827+0000 7fc5fe59c700 1 -- 172.21.15.46:0/1111393208 learned_addr learned my addr 172.21.15.46:0/1111393208 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:49.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.828+0000 7fc5fe59c700 1 -- 172.21.15.46:0/1111393208 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc600101ab0 con 0x7fc600101020 2023-06-23T01:01:49.400 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.828+0000 7fc5fe59c700 1 --2- 172.21.15.46:0/1111393208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc600101430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000bb80 tx=0x7fc5f000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c06d63d99a813d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:49.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.828+0000 7fc5fd59a700 1 -- 172.21.15.46:0/1111393208 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5f000dba0 con 0x7fc600101020 2023-06-23T01:01:49.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.828+0000 7fc5fd59a700 1 -- 172.21.15.46:0/1111393208 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc5f000dd00 con 0x7fc600101020 2023-06-23T01:01:49.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.828+0000 7fc5fd59a700 1 -- 172.21.15.46:0/1111393208 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5f00185e0 con 0x7fc600101020 2023-06-23T01:01:49.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.828+0000 7fc604bb6700 1 -- 172.21.15.46:0/1111393208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 msgr2=0x7fc600101430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:49.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.829+0000 7fc604bb6700 1 --2- 172.21.15.46:0/1111393208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc600101430 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000bb80 tx=0x7fc5f000b4f0 comp rx=0 tx=0).stop 2023-06-23T01:01:49.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.829+0000 7fc604bb6700 1 -- 172.21.15.46:0/1111393208 shutdown_connections 2023-06-23T01:01:49.401 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.829+0000 7fc604bb6700 1 --2- 172.21.15.46:0/1111393208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc600101430 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:49.402 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.829+0000 7fc604bb6700 1 -- 172.21.15.46:0/1111393208 >> 172.21.15.46:0/1111393208 conn(0x7fc6000fc7e0 msgr2=0x7fc6000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:49.402 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.829+0000 7fc604bb6700 1 -- 172.21.15.46:0/1111393208 shutdown_connections 2023-06-23T01:01:49.402 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.829+0000 7fc604bb6700 1 -- 172.21.15.46:0/1111393208 wait complete. 2023-06-23T01:01:49.402 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.829+0000 7fc604bb6700 1 Processor -- start 2023-06-23T01:01:49.402 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.830+0000 7fc604bb6700 1 -- start start 2023-06-23T01:01:49.402 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.830+0000 7fc604bb6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc60010eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:49.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.830+0000 7fc604bb6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60010f0b0 con 0x7fc600101020 2023-06-23T01:01:49.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.830+0000 7fc5fe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc60010eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:49.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.830+0000 7fc5fe59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc60010eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47668/0 (socket says 172.21.15.46:47668) 2023-06-23T01:01:49.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.830+0000 7fc5fe59c700 1 -- 172.21.15.46:0/3135666210 learned_addr learned my addr 172.21.15.46:0/3135666210 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:49.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.830+0000 7fc5fe59c700 1 -- 172.21.15.46:0/3135666210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f000b790 con 0x7fc600101020 2023-06-23T01:01:49.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.830+0000 7fc5fe59c700 1 --2- 172.21.15.46:0/3135666210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc60010eb70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000b760 tx=0x7fc5f000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:49.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.831+0000 7fc5eeffd700 1 -- 172.21.15.46:0/3135666210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5f0010750 con 0x7fc600101020 2023-06-23T01:01:49.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.831+0000 7fc604bb6700 1 -- 172.21.15.46:0/3135666210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60010f2b0 con 0x7fc600101020 2023-06-23T01:01:49.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.831+0000 7fc5eeffd700 1 -- 172.21.15.46:0/3135666210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc5f001d040 con 0x7fc600101020 2023-06-23T01:01:49.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.831+0000 7fc5eeffd700 1 -- 172.21.15.46:0/3135666210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5f0020390 con 0x7fc600101020 2023-06-23T01:01:49.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.831+0000 7fc604bb6700 1 -- 172.21.15.46:0/3135666210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60010ff50 con 0x7fc600101020 2023-06-23T01:01:49.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.831+0000 7fc5eeffd700 1 -- 172.21.15.46:0/3135666210 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 43219+0+0 (secure 0 0 0) 0x7fc5f0020650 con 0x7fc600101020 2023-06-23T01:01:49.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.832+0000 7fc5eeffd700 1 --2- 172.21.15.46:0/3135666210 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fc5e8036560 0x7fc5e8038a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:49.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.832+0000 7fc5eeffd700 1 -- 172.21.15.46:0/3135666210 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc5f0014070 con 0x7fc600101020 2023-06-23T01:01:49.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.832+0000 7fc5fdd9b700 1 --2- 172.21.15.46:0/3135666210 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fc5e8036560 0x7fc5e8038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:49.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.832+0000 7fc604bb6700 1 -- 172.21.15.46:0/3135666210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60010f740 con 0x7fc600101020 2023-06-23T01:01:49.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.832+0000 7fc5fdd9b700 1 --2- 172.21.15.46:0/3135666210 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fc5e8036560 0x7fc5e8038a10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009990 tx=0x7fc5f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:49.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:47.837+0000 7fc5eeffd700 1 -- 172.21.15.46:0/3135666210 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fc5f0028030 con 0x7fc600101020 2023-06-23T01:01:49.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.090+0000 7fc604bb6700 1 -- 172.21.15.46:0/3135666210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fc6001030a0 con 0x7fc600101020 2023-06-23T01:01:49.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.937+0000 7fc5eeffd700 1 -- 172.21.15.46:0/3135666210 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 9) v1 ==== 43232+0+0 (secure 0 0 0) 0x7fc5f002d7c0 con 0x7fc600101020 2023-06-23T01:01:49.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.938+0000 7fc5eeffd700 1 -- 172.21.15.46:0/3135666210 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fc5f002b360 con 0x7fc600101020 2023-06-23T01:01:49.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.940+0000 7fc604bb6700 1 -- 172.21.15.46:0/3135666210 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fc5e8036560 msgr2=0x7fc5e8038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:49.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.940+0000 7fc604bb6700 1 --2- 172.21.15.46:0/3135666210 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fc5e8036560 0x7fc5e8038a10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009990 tx=0x7fc5f4006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:49.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.941+0000 7fc604bb6700 1 -- 172.21.15.46:0/3135666210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 msgr2=0x7fc60010eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:49.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.941+0000 7fc604bb6700 1 --2- 172.21.15.46:0/3135666210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc60010eb70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000b760 tx=0x7fc5f000b4f0 comp rx=0 tx=0).stop 2023-06-23T01:01:49.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.941+0000 7fc604bb6700 1 -- 172.21.15.46:0/3135666210 shutdown_connections 2023-06-23T01:01:49.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.941+0000 7fc604bb6700 1 --2- 172.21.15.46:0/3135666210 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fc5e8036560 0x7fc5e8038a10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:49.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.941+0000 7fc604bb6700 1 --2- 172.21.15.46:0/3135666210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc600101020 0x7fc60010eb70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:49.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.941+0000 7fc604bb6700 1 -- 172.21.15.46:0/3135666210 >> 172.21.15.46:0/3135666210 conn(0x7fc6000fc7e0 msgr2=0x7fc600105540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:49.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.941+0000 7fc604bb6700 1 -- 172.21.15.46:0/3135666210 shutdown_connections 2023-06-23T01:01:49.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:48.941+0000 7fc604bb6700 1 -- 172.21.15.46:0/3135666210 wait complete. 2023-06-23T01:01:50.283 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:49 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3135666210' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-06-23T01:01:50.283 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:49 smithi046 ceph-mon[84949]: mgrmap e9: x(active, since 16s) 2023-06-23T01:01:50.546 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:50 smithi046 conmon[85362]: 2023-06-23T01:01:50.278+0000 7fb2cdf601c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:01:50.546 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:50 smithi046 conmon[85362]: 2023-06-23T01:01:50.449+0000 7fb2cdf601c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:01:50.547 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:50 smithi046 conmon[85362]: 2023-06-23T01:01:50.543+0000 7fb2cdf601c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:01:50.763 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2023-06-23T01:01:50.764 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-06-23T01:01:50.764 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2023-06-23T01:01:50.765 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-06-23T01:01:50.765 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-06-23T01:01:50.765 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2023-06-23T01:01:50.765 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.088+0000 7fa71043b700 1 Processor -- start 2023-06-23T01:01:50.766 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.088+0000 7fa71043b700 1 -- start start 2023-06-23T01:01:50.766 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.089+0000 7fa71043b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa708102a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:50.766 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.089+0000 7fa71043b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa708102fd0 con 0x7fa708102680 2023-06-23T01:01:50.767 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.089+0000 7fa70e1d7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa708102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:50.767 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.089+0000 7fa70e1d7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa708102a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47698/0 (socket says 172.21.15.46:47698) 2023-06-23T01:01:50.767 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.089+0000 7fa70e1d7700 1 -- 172.21.15.46:0/866146571 learned_addr learned my addr 172.21.15.46:0/866146571 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:50.768 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.089+0000 7fa70e1d7700 1 -- 172.21.15.46:0/866146571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa708103110 con 0x7fa708102680 2023-06-23T01:01:50.768 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.089+0000 7fa70e1d7700 1 --2- 172.21.15.46:0/866146571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa708102a90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa70400bb80 tx=0x7fa70400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2039a2c8cef64bcf server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:50.768 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.090+0000 7fa70d1d5700 1 -- 172.21.15.46:0/866146571 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa70400dba0 con 0x7fa708102680 2023-06-23T01:01:50.768 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.090+0000 7fa70d1d5700 1 -- 172.21.15.46:0/866146571 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa70400dd00 con 0x7fa708102680 2023-06-23T01:01:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.090+0000 7fa70d1d5700 1 -- 172.21.15.46:0/866146571 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7040185e0 con 0x7fa708102680 2023-06-23T01:01:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.090+0000 7fa71043b700 1 -- 172.21.15.46:0/866146571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 msgr2=0x7fa708102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.090+0000 7fa71043b700 1 --2- 172.21.15.46:0/866146571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa708102a90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa70400bb80 tx=0x7fa70400b4f0 comp rx=0 tx=0).stop 2023-06-23T01:01:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.090+0000 7fa71043b700 1 -- 172.21.15.46:0/866146571 shutdown_connections 2023-06-23T01:01:50.770 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.090+0000 7fa71043b700 1 --2- 172.21.15.46:0/866146571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa708102a90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:50.770 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.090+0000 7fa71043b700 1 -- 172.21.15.46:0/866146571 >> 172.21.15.46:0/866146571 conn(0x7fa7080fbfa0 msgr2=0x7fa7080fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:50.770 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.090+0000 7fa71043b700 1 -- 172.21.15.46:0/866146571 shutdown_connections 2023-06-23T01:01:50.770 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.090+0000 7fa71043b700 1 -- 172.21.15.46:0/866146571 wait complete. 2023-06-23T01:01:50.770 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.091+0000 7fa71043b700 1 Processor -- start 2023-06-23T01:01:50.771 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.091+0000 7fa71043b700 1 -- start start 2023-06-23T01:01:50.771 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.091+0000 7fa71043b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa70810eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:50.771 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.091+0000 7fa71043b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70810f0d0 con 0x7fa708102680 2023-06-23T01:01:50.772 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.091+0000 7fa70e1d7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa70810eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:50.772 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.091+0000 7fa70e1d7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa70810eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47708/0 (socket says 172.21.15.46:47708) 2023-06-23T01:01:50.772 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.091+0000 7fa70e1d7700 1 -- 172.21.15.46:0/676342013 learned_addr learned my addr 172.21.15.46:0/676342013 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:50.772 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.092+0000 7fa70e1d7700 1 -- 172.21.15.46:0/676342013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70400b790 con 0x7fa708102680 2023-06-23T01:01:50.772 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.092+0000 7fa70e1d7700 1 --2- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa70810eb90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa704006d40 tx=0x7fa70400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:50.772 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.092+0000 7fa6faffd700 1 -- 172.21.15.46:0/676342013 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa704027070 con 0x7fa708102680 2023-06-23T01:01:50.773 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.092+0000 7fa71043b700 1 -- 172.21.15.46:0/676342013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70810f2d0 con 0x7fa708102680 2023-06-23T01:01:50.773 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.092+0000 7fa6faffd700 1 -- 172.21.15.46:0/676342013 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa70401d050 con 0x7fa708102680 2023-06-23T01:01:50.773 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.092+0000 7fa6faffd700 1 -- 172.21.15.46:0/676342013 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa704023cd0 con 0x7fa708102680 2023-06-23T01:01:50.773 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.092+0000 7fa71043b700 1 -- 172.21.15.46:0/676342013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70810f6e0 con 0x7fa708102680 2023-06-23T01:01:50.773 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.093+0000 7fa71043b700 1 -- 172.21.15.46:0/676342013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70804b970 con 0x7fa708102680 2023-06-23T01:01:50.773 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.093+0000 7fa6faffd700 1 -- 172.21.15.46:0/676342013 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) v1 ==== 43232+0+0 (secure 0 0 0) 0x7fa70402c410 con 0x7fa708102680 2023-06-23T01:01:50.773 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.093+0000 7fa6faffd700 1 --2- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa6f40365c0 0x7fa6f4038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:50.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.093+0000 7fa6faffd700 1 -- 172.21.15.46:0/676342013 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa704014070 con 0x7fa708102680 2023-06-23T01:01:50.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.093+0000 7fa70d9d6700 1 -- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa6f40365c0 msgr2=0x7fa6f4038a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2331157358 2023-06-23T01:01:50.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.093+0000 7fa70d9d6700 1 --2- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa6f40365c0 0x7fa6f4038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:01:50.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.102+0000 7fa6faffd700 1 -- 172.21.15.46:0/676342013 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fa70400e360 con 0x7fa708102680 2023-06-23T01:01:50.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.294+0000 7fa70d9d6700 1 -- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa6f40365c0 msgr2=0x7fa6f4038a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2331157358 2023-06-23T01:01:50.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.294+0000 7fa70d9d6700 1 --2- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa6f40365c0 0x7fa6f4038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-23T01:01:50.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.353+0000 7fa71043b700 1 -- 172.21.15.46:0/676342013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa7081082a0 con 0x7fa708102680 2023-06-23T01:01:50.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.353+0000 7fa6faffd700 1 -- 172.21.15.46:0/676342013 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fa70404c650 con 0x7fa708102680 2023-06-23T01:01:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.355+0000 7fa71043b700 1 -- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa6f40365c0 msgr2=0x7fa6f4038a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-23T01:01:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.355+0000 7fa71043b700 1 --2- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa6f40365c0 0x7fa6f4038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.355+0000 7fa71043b700 1 -- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 msgr2=0x7fa70810eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.355+0000 7fa71043b700 1 --2- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa70810eb90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa704006d40 tx=0x7fa70400b4f0 comp rx=0 tx=0).stop 2023-06-23T01:01:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.356+0000 7fa71043b700 1 -- 172.21.15.46:0/676342013 shutdown_connections 2023-06-23T01:01:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.356+0000 7fa71043b700 1 --2- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7fa6f40365c0 0x7fa6f4038a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.356+0000 7fa71043b700 1 --2- 172.21.15.46:0/676342013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa708102680 0x7fa70810eb90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.356+0000 7fa71043b700 1 -- 172.21.15.46:0/676342013 >> 172.21.15.46:0/676342013 conn(0x7fa7080fbfa0 msgr2=0x7fa708105620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.356+0000 7fa71043b700 1 -- 172.21.15.46:0/676342013 shutdown_connections 2023-06-23T01:01:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:50.356+0000 7fa71043b700 1 -- 172.21.15.46:0/676342013 wait complete. 2023-06-23T01:01:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for the mgr to restart... 2023-06-23T01:01:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr epoch 9... 2023-06-23T01:01:50.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:50 smithi046 conmon[85362]: 2023-06-23T01:01:50.773+0000 7fb2cdf601c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:01:51.258 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:50 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/676342013' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-06-23T01:01:51.259 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:51 smithi046 conmon[85362]: 2023-06-23T01:01:50.998+0000 7fb2cdf601c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:01:51.259 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:51 smithi046 conmon[85362]: 2023-06-23T01:01:51.168+0000 7fb2cdf601c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:01:51.259 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:51 smithi046 conmon[85362]: 2023-06-23T01:01:51.255+0000 7fb2cdf601c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:01:52.318 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:51 smithi046 conmon[85362]: 2023-06-23T01:01:51.868+0000 7fb2cdf601c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:01:52.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:52 smithi046 conmon[85362]: 2023-06-23T01:01:52.375+0000 7fb2cdf601c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:01:52.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:52 smithi046 conmon[85362]: 2023-06-23T01:01:52.472+0000 7fb2cdf601c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:01:52.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:52 smithi046 conmon[85362]: 2023-06-23T01:01:52.570+0000 7fb2cdf601c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:01:53.477 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:53 smithi046 conmon[85362]: 2023-06-23T01:01:53.200+0000 7fb2cdf601c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:01:53.477 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:53 smithi046 conmon[85362]: 2023-06-23T01:01:53.378+0000 7fb2cdf601c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:01:53.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:53 smithi046 conmon[85362]: 2023-06-23T01:01:53.473+0000 7fb2cdf601c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:01:53.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:53 smithi046 conmon[85362]: 2023-06-23T01:01:53.609+0000 7fb2cdf601c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:01:53.818 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:53 smithi046 conmon[85362]: 2023-06-23T01:01:53.707+0000 7fb2cdf601c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:01:54.981 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:54 smithi046 conmon[85362]: 2023-06-23T01:01:54.633+0000 7fb2cdf601c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:01:54.981 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:54 smithi046 conmon[85362]: 2023-06-23T01:01:54.726+0000 7fb2cdf601c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:01:54.981 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:54 smithi046 conmon[85362]: 2023-06-23T01:01:54.820+0000 7fb2cdf601c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:01:54.981 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:54 smithi046 conmon[85362]: 2023-06-23T01:01:54.977+0000 7fb2cdf601c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:01:55.318 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:55 smithi046 conmon[85362]: 2023-06-23T01:01:55.081+0000 7fb2cdf601c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:01:55.775 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:55 smithi046 conmon[85362]: 2023-06-23T01:01:55.491+0000 7fb2cdf601c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:01:55.775 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:55 smithi046 conmon[85362]: 2023-06-23T01:01:55.585+0000 7fb2cdf601c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:01:55.775 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:55 smithi046 conmon[85362]: 2023-06-23T01:01:55.679+0000 7fb2cdf601c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:01:55.775 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:01:55 smithi046 conmon[85362]: 2023-06-23T01:01:55.771+0000 7fb2cdf601c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:01:56.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: Active manager daemon x restarted 2023-06-23T01:01:56.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: Activating manager daemon x 2023-06-23T01:01:56.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: osdmap e3: 0 total, 0 up, 0 in 2023-06-23T01:01:56.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: mgrmap e10: x(active, starting, since 0.00326421s) 2023-06-23T01:01:56.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:01:56.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-06-23T01:01:56.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-06-23T01:01:56.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-06-23T01:01:56.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-06-23T01:01:56.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: Manager daemon x is now available 2023-06-23T01:01:56.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:01:56.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:55 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:01:57.153 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-06-23T01:01:57.153 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-06-23T01:01:57.154 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:56 smithi046 ceph-mon[84949]: mgrmap e11: x(active, since 1.00568s) 2023-06-23T01:01:57.221 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2023-06-23T01:01:57.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-06-23T01:01:57.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "initialized": true 2023-06-23T01:01:57.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2023-06-23T01:01:57.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.206+0000 7f760be90700 1 Processor -- start 2023-06-23T01:01:57.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.206+0000 7f760be90700 1 -- start start 2023-06-23T01:01:57.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.206+0000 7f760be90700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604107190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:57.222 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.206+0000 7f760be90700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604102b80 con 0x7f7604070db0 2023-06-23T01:01:57.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.206+0000 7f7609c2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:57.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.206+0000 7f7609c2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604107190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47718/0 (socket says 172.21.15.46:47718) 2023-06-23T01:01:57.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.206+0000 7f7609c2c700 1 -- 172.21.15.46:0/960539473 learned_addr learned my addr 172.21.15.46:0/960539473 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:57.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.207+0000 7f7609c2c700 1 -- 172.21.15.46:0/960539473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76041076d0 con 0x7f7604070db0 2023-06-23T01:01:57.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.207+0000 7f7609c2c700 1 --2- 172.21.15.46:0/960539473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604107190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f76000098e0 tx=0x7f7600009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70185d62ded662d8 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:57.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.207+0000 7f7608c2a700 1 -- 172.21.15.46:0/960539473 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7600009e80 con 0x7f7604070db0 2023-06-23T01:01:57.223 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.207+0000 7f7608c2a700 1 -- 172.21.15.46:0/960539473 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f760000bd00 con 0x7f7604070db0 2023-06-23T01:01:57.224 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.207+0000 7f7608c2a700 1 -- 172.21.15.46:0/960539473 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7600011c00 con 0x7f7604070db0 2023-06-23T01:01:57.224 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.208+0000 7f760be90700 1 -- 172.21.15.46:0/960539473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 msgr2=0x7f7604107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:57.224 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.208+0000 7f760be90700 1 --2- 172.21.15.46:0/960539473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604107190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f76000098e0 tx=0x7f7600009250 comp rx=0 tx=0).stop 2023-06-23T01:01:57.224 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.208+0000 7f760be90700 1 -- 172.21.15.46:0/960539473 shutdown_connections 2023-06-23T01:01:57.224 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.208+0000 7f760be90700 1 --2- 172.21.15.46:0/960539473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604107190 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:57.224 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.208+0000 7f760be90700 1 -- 172.21.15.46:0/960539473 >> 172.21.15.46:0/960539473 conn(0x7f76040fc800 msgr2=0x7f76040fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:57.224 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.208+0000 7f760be90700 1 -- 172.21.15.46:0/960539473 shutdown_connections 2023-06-23T01:01:57.225 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.208+0000 7f760be90700 1 -- 172.21.15.46:0/960539473 wait complete. 2023-06-23T01:01:57.225 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.208+0000 7f760be90700 1 Processor -- start 2023-06-23T01:01:57.225 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.208+0000 7f760be90700 1 -- start start 2023-06-23T01:01:57.225 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f760be90700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:57.226 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f760be90700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604113560 con 0x7f7604070db0 2023-06-23T01:01:57.226 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f7609c2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:57.226 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f7609c2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604113020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47732/0 (socket says 172.21.15.46:47732) 2023-06-23T01:01:57.227 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f7609c2c700 1 -- 172.21.15.46:0/1000731193 learned_addr learned my addr 172.21.15.46:0/1000731193 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:57.227 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f7609c2c700 1 -- 172.21.15.46:0/1000731193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76000094f0 con 0x7f7604070db0 2023-06-23T01:01:57.227 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f7609c2c700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604113020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7600011650 tx=0x7f7600009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:57.227 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f760001a760 con 0x7f7604070db0 2023-06-23T01:01:57.228 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f760be90700 1 -- 172.21.15.46:0/1000731193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7604113760 con 0x7f7604070db0 2023-06-23T01:01:57.228 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f760001ad60 con 0x7f7604070db0 2023-06-23T01:01:57.228 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7600022b60 con 0x7f7604070db0 2023-06-23T01:01:57.229 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.209+0000 7f760be90700 1 -- 172.21.15.46:0/1000731193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7604113b70 con 0x7f7604070db0 2023-06-23T01:01:57.229 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.210+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) v1 ==== 43232+0+0 (secure 0 0 0) 0x7f7600022500 con 0x7f7604070db0 2023-06-23T01:01:57.229 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.211+0000 7f75fa7fc700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 0x7f75f0038a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:57.229 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.211+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 --> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75f0039180 con 0x7f75f00365c0 2023-06-23T01:01:57.229 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.211+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f760004a500 con 0x7f7604070db0 2023-06-23T01:01:57.230 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.211+0000 7f760942b700 1 -- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 msgr2=0x7f75f0038a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2331157358 2023-06-23T01:01:57.230 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.211+0000 7f760942b700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 0x7f75f0038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:01:57.230 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.412+0000 7f760942b700 1 -- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 msgr2=0x7f75f0038a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2331157358 2023-06-23T01:01:57.230 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.412+0000 7f760942b700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 0x7f75f0038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-23T01:01:57.231 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.813+0000 7f760942b700 1 -- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 msgr2=0x7f75f0038a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2331157358 2023-06-23T01:01:57.231 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:51.813+0000 7f760942b700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 0x7f75f0038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-23T01:01:57.231 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:52.614+0000 7f760942b700 1 -- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 msgr2=0x7f75f0038a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2331157358 2023-06-23T01:01:57.231 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:52.614+0000 7f760942b700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 0x7f75f0038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-23T01:01:57.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:54.215+0000 7f760942b700 1 -- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 msgr2=0x7f75f0038a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2331157358 2023-06-23T01:01:57.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:54.215+0000 7f760942b700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 0x7f75f0038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-06-23T01:01:57.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:55.787+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 10) v1 ==== 42999+0+0 (secure 0 0 0) 0x7f760002c960 con 0x7f7604070db0 2023-06-23T01:01:57.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:55.787+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 msgr2=0x7f75f0038a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-23T01:01:57.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:55.787+0000 7f75fa7fc700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 0x7f75f0038a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:57.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.788+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 11) v1 ==== 43126+0+0 (secure 0 0 0) 0x7f760001aa90 con 0x7f7604070db0 2023-06-23T01:01:57.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.789+0000 7f75fa7fc700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f75f0039fd0 0x7f75f003c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:57.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.789+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f75f0039180 con 0x7f75f0039fd0 2023-06-23T01:01:57.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.789+0000 7f760942b700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f75f0039fd0 0x7f75f003c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:57.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.790+0000 7f760942b700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f75f0039fd0 0x7f75f003c3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f75f40039d0 tx=0x7f75f40092f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:57.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.790+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f75f0039180 con 0x7f75f0039fd0 2023-06-23T01:01:57.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.801+0000 7f760be90700 1 -- 172.21.15.46:0/1000731193 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f760404b970 con 0x7f75f0039fd0 2023-06-23T01:01:57.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.801+0000 7f75fa7fc700 1 -- 172.21.15.46:0/1000731193 <== mgr.14152 v2:172.21.15.46:6800/1578446983 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f760404b970 con 0x7f75f0039fd0 2023-06-23T01:01:57.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.801+0000 7f760be90700 1 -- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f75f0039fd0 msgr2=0x7f75f003c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:57.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.801+0000 7f760be90700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f75f0039fd0 0x7f75f003c3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f75f40039d0 tx=0x7f75f40092f0 comp rx=0 tx=0).stop 2023-06-23T01:01:57.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.801+0000 7f760be90700 1 -- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 msgr2=0x7f7604113020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:57.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.801+0000 7f760be90700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604113020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7600011650 tx=0x7f7600009250 comp rx=0 tx=0).stop 2023-06-23T01:01:57.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.801+0000 7f760be90700 1 -- 172.21.15.46:0/1000731193 shutdown_connections 2023-06-23T01:01:57.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.801+0000 7f760be90700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f75f0039fd0 0x7f75f003c3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:57.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.801+0000 7f760be90700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7604070db0 0x7f7604113020 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:57.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.801+0000 7f760be90700 1 --2- 172.21.15.46:0/1000731193 >> [v2:172.21.15.46:6800/2331157358,v1:172.21.15.46:6801/2331157358] conn(0x7f75f00365c0 0x7f75f0038a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:57.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.801+0000 7f760be90700 1 -- 172.21.15.46:0/1000731193 >> 172.21.15.46:0/1000731193 conn(0x7f76040fc800 msgr2=0x7f76040fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:57.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.802+0000 7f760be90700 1 -- 172.21.15.46:0/1000731193 shutdown_connections 2023-06-23T01:01:57.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:56.802+0000 7f760be90700 1 -- 172.21.15.46:0/1000731193 wait complete. 2023-06-23T01:01:57.235 INFO:teuthology.orchestra.run.smithi046.stdout:mgr epoch 9 is available 2023-06-23T01:01:57.235 INFO:teuthology.orchestra.run.smithi046.stdout:Generating a dashboard self-signed certificate... 2023-06-23T01:01:58.132 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:57 smithi046 ceph-mon[84949]: [23/Jun/2023:01:01:56] ENGINE Bus STARTING 2023-06-23T01:01:58.132 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:01:58.132 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:01:58.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-06-23T01:01:58.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.817+0000 7f3cfc680700 1 Processor -- start 2023-06-23T01:01:58.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.817+0000 7f3cfc680700 1 -- start start 2023-06-23T01:01:58.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.817+0000 7f3cfc680700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4076160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:58.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.817+0000 7f3cfc680700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4078060 con 0x7f3cf4077c50 2023-06-23T01:01:58.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.818+0000 7f3cfa41c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4076160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:58.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.818+0000 7f3cfa41c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4076160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37660/0 (socket says 172.21.15.46:37660) 2023-06-23T01:01:58.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.818+0000 7f3cfa41c700 1 -- 172.21.15.46:0/3300368240 learned_addr learned my addr 172.21.15.46:0/3300368240 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:58.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.818+0000 7f3cfa41c700 1 -- 172.21.15.46:0/3300368240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf40766a0 con 0x7f3cf4077c50 2023-06-23T01:01:58.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.818+0000 7f3cfa41c700 1 --2- 172.21.15.46:0/3300368240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4076160 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009e50 tx=0x7f3ce4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df514c4ea78ff165 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:58.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.818+0000 7f3cf941a700 1 -- 172.21.15.46:0/3300368240 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ce400e070 con 0x7f3cf4077c50 2023-06-23T01:01:58.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.818+0000 7f3cf941a700 1 -- 172.21.15.46:0/3300368240 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3ce4012430 con 0x7f3cf4077c50 2023-06-23T01:01:58.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.818+0000 7f3cf941a700 1 -- 172.21.15.46:0/3300368240 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ce40092e0 con 0x7f3cf4077c50 2023-06-23T01:01:58.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.819+0000 7f3cfc680700 1 -- 172.21.15.46:0/3300368240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 msgr2=0x7f3cf4076160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:58.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.819+0000 7f3cfc680700 1 --2- 172.21.15.46:0/3300368240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4076160 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009e50 tx=0x7f3ce4009840 comp rx=0 tx=0).stop 2023-06-23T01:01:58.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.819+0000 7f3cfc680700 1 -- 172.21.15.46:0/3300368240 shutdown_connections 2023-06-23T01:01:58.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.820+0000 7f3cfc680700 1 --2- 172.21.15.46:0/3300368240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4076160 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:58.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.820+0000 7f3cfc680700 1 -- 172.21.15.46:0/3300368240 >> 172.21.15.46:0/3300368240 conn(0x7f3cf40fd920 msgr2=0x7f3cf40ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:58.409 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.820+0000 7f3cfc680700 1 -- 172.21.15.46:0/3300368240 shutdown_connections 2023-06-23T01:01:58.409 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.820+0000 7f3cfc680700 1 -- 172.21.15.46:0/3300368240 wait complete. 2023-06-23T01:01:58.409 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.820+0000 7f3cfc680700 1 Processor -- start 2023-06-23T01:01:58.409 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.820+0000 7f3cfc680700 1 -- start start 2023-06-23T01:01:58.409 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.821+0000 7f3cfc680700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:58.409 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.821+0000 7f3cfc680700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf41135b0 con 0x7f3cf4077c50 2023-06-23T01:01:58.409 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.821+0000 7f3cfa41c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:58.410 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.821+0000 7f3cfa41c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4113070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37668/0 (socket says 172.21.15.46:37668) 2023-06-23T01:01:58.410 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.821+0000 7f3cfa41c700 1 -- 172.21.15.46:0/2224355413 learned_addr learned my addr 172.21.15.46:0/2224355413 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:58.410 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.821+0000 7f3cfa41c700 1 -- 172.21.15.46:0/2224355413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce4009ae0 con 0x7f3cf4077c50 2023-06-23T01:01:58.410 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.821+0000 7f3cfa41c700 1 --2- 172.21.15.46:0/2224355413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4113070 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4012c50 tx=0x7f3ce4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:58.410 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.822+0000 7f3cf2ffd700 1 -- 172.21.15.46:0/2224355413 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ce400e030 con 0x7f3cf4077c50 2023-06-23T01:01:58.410 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.822+0000 7f3cf2ffd700 1 -- 172.21.15.46:0/2224355413 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3ce40092e0 con 0x7f3cf4077c50 2023-06-23T01:01:58.410 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.822+0000 7f3cf2ffd700 1 -- 172.21.15.46:0/2224355413 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3ce40075c0 con 0x7f3cf4077c50 2023-06-23T01:01:58.410 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.822+0000 7f3cfc680700 1 -- 172.21.15.46:0/2224355413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf41137b0 con 0x7f3cf4077c50 2023-06-23T01:01:58.411 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.822+0000 7f3cfc680700 1 -- 172.21.15.46:0/2224355413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf4114450 con 0x7f3cf4077c50 2023-06-23T01:01:58.411 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.823+0000 7f3cf2ffd700 1 -- 172.21.15.46:0/2224355413 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 11) v1 ==== 43126+0+0 (secure 0 0 0) 0x7f3ce4019070 con 0x7f3cf4077c50 2023-06-23T01:01:58.411 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.823+0000 7f3cfc680700 1 -- 172.21.15.46:0/2224355413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf405f060 con 0x7f3cf4077c50 2023-06-23T01:01:58.411 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.823+0000 7f3cf2ffd700 1 --2- 172.21.15.46:0/2224355413 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3cdc036530 0x7f3cdc0389e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:58.411 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.823+0000 7f3cf2ffd700 1 -- 172.21.15.46:0/2224355413 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3ce4049890 con 0x7f3cf4077c50 2023-06-23T01:01:58.411 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.823+0000 7f3cf9c1b700 1 --2- 172.21.15.46:0/2224355413 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3cdc036530 0x7f3cdc0389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:58.411 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.824+0000 7f3cf9c1b700 1 --2- 172.21.15.46:0/2224355413 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3cdc036530 0x7f3cdc0389e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009990 tx=0x7f3ce8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:58.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:57.828+0000 7f3cf2ffd700 1 -- 172.21.15.46:0/2224355413 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f3ce4019320 con 0x7f3cf4077c50 2023-06-23T01:01:58.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.035+0000 7f3cfc680700 1 -- 172.21.15.46:0/2224355413 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cf40fed70 con 0x7f3cdc036530 2023-06-23T01:01:58.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.105+0000 7f3cf2ffd700 1 -- 172.21.15.46:0/2224355413 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3cf40fed70 con 0x7f3cdc036530 2023-06-23T01:01:58.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.107+0000 7f3cfc680700 1 -- 172.21.15.46:0/2224355413 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3cdc036530 msgr2=0x7f3cdc0389e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:58.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.107+0000 7f3cfc680700 1 --2- 172.21.15.46:0/2224355413 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3cdc036530 0x7f3cdc0389e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009990 tx=0x7f3ce8006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:58.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.107+0000 7f3cfc680700 1 -- 172.21.15.46:0/2224355413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 msgr2=0x7f3cf4113070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:58.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.107+0000 7f3cfc680700 1 --2- 172.21.15.46:0/2224355413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4113070 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4012c50 tx=0x7f3ce4009840 comp rx=0 tx=0).stop 2023-06-23T01:01:58.413 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.107+0000 7f3cfc680700 1 -- 172.21.15.46:0/2224355413 shutdown_connections 2023-06-23T01:01:58.413 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.107+0000 7f3cfc680700 1 --2- 172.21.15.46:0/2224355413 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3cdc036530 0x7f3cdc0389e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:58.413 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.107+0000 7f3cfc680700 1 --2- 172.21.15.46:0/2224355413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3cf4077c50 0x7f3cf4113070 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:58.413 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.107+0000 7f3cfc680700 1 -- 172.21.15.46:0/2224355413 >> 172.21.15.46:0/2224355413 conn(0x7f3cf40fd920 msgr2=0x7f3cf40fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:58.413 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.108+0000 7f3cfc680700 1 -- 172.21.15.46:0/2224355413 shutdown_connections 2023-06-23T01:01:58.414 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.108+0000 7f3cfc680700 1 -- 172.21.15.46:0/2224355413 wait complete. 2023-06-23T01:01:58.414 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial admin user... 2023-06-23T01:01:59.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:58 smithi046 ceph-mon[84949]: [23/Jun/2023:01:01:57] ENGINE Serving on https://172.21.15.46:7150 2023-06-23T01:01:59.369 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:58 smithi046 ceph-mon[84949]: [23/Jun/2023:01:01:57] ENGINE Bus STARTED 2023-06-23T01:01:59.370 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:01:59.370 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:01:59.370 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:01:59.370 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:58 smithi046 ceph-mon[84949]: mgrmap e12: x(active, since 2s) 2023-06-23T01:01:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$.6ZQj6ykrRSwRtMATvkCyembb8.BgU6Sh69oXxIfo.xUwc.j9q0.O", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1687482119, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-06-23T01:01:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.861+0000 7f613e1b7700 1 Processor -- start 2023-06-23T01:01:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.861+0000 7f613e1b7700 1 -- start start 2023-06-23T01:01:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.861+0000 7f613e1b7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f6138101490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.861+0000 7f613e1b7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61381019d0 con 0x7f6138101080 2023-06-23T01:01:59.947 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.861+0000 7f61377fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f6138101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.861+0000 7f61377fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f6138101490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37672/0 (socket says 172.21.15.46:37672) 2023-06-23T01:01:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.861+0000 7f61377fe700 1 -- 172.21.15.46:0/1006662415 learned_addr learned my addr 172.21.15.46:0/1006662415 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.862+0000 7f61377fe700 1 -- 172.21.15.46:0/1006662415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138101b10 con 0x7f6138101080 2023-06-23T01:01:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.862+0000 7f61377fe700 1 --2- 172.21.15.46:0/1006662415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f6138101490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f61280098e0 tx=0x7f6128009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed60154748ad4125 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.862+0000 7f61367fc700 1 -- 172.21.15.46:0/1006662415 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6128009e80 con 0x7f6138101080 2023-06-23T01:01:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.862+0000 7f61367fc700 1 -- 172.21.15.46:0/1006662415 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f612800bd00 con 0x7f6138101080 2023-06-23T01:01:59.948 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.862+0000 7f61367fc700 1 -- 172.21.15.46:0/1006662415 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6128011c00 con 0x7f6138101080 2023-06-23T01:01:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.862+0000 7f613e1b7700 1 -- 172.21.15.46:0/1006662415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 msgr2=0x7f6138101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.862+0000 7f613e1b7700 1 --2- 172.21.15.46:0/1006662415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f6138101490 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f61280098e0 tx=0x7f6128009250 comp rx=0 tx=0).stop 2023-06-23T01:01:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.863+0000 7f613e1b7700 1 -- 172.21.15.46:0/1006662415 shutdown_connections 2023-06-23T01:01:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.863+0000 7f613e1b7700 1 --2- 172.21.15.46:0/1006662415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f6138101490 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.863+0000 7f613e1b7700 1 -- 172.21.15.46:0/1006662415 >> 172.21.15.46:0/1006662415 conn(0x7f61380fc800 msgr2=0x7f61380fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.863+0000 7f613e1b7700 1 -- 172.21.15.46:0/1006662415 shutdown_connections 2023-06-23T01:01:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.863+0000 7f613e1b7700 1 -- 172.21.15.46:0/1006662415 wait complete. 2023-06-23T01:01:59.949 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.863+0000 7f613e1b7700 1 Processor -- start 2023-06-23T01:01:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.863+0000 7f613e1b7700 1 -- start start 2023-06-23T01:01:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.863+0000 7f613e1b7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f613810ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.863+0000 7f613e1b7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613810f140 con 0x7f6138101080 2023-06-23T01:01:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.864+0000 7f61377fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f613810ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.864+0000 7f61377fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f613810ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37674/0 (socket says 172.21.15.46:37674) 2023-06-23T01:01:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.864+0000 7f61377fe700 1 -- 172.21.15.46:0/1920007829 learned_addr learned my addr 172.21.15.46:0/1920007829 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:01:59.950 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.864+0000 7f61377fe700 1 -- 172.21.15.46:0/1920007829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61280094f0 con 0x7f6138101080 2023-06-23T01:01:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.864+0000 7f61377fe700 1 --2- 172.21.15.46:0/1920007829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f613810ec00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6128011650 tx=0x7f6128009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.864+0000 7f611ffff700 1 -- 172.21.15.46:0/1920007829 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f612801a670 con 0x7f6138101080 2023-06-23T01:01:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.864+0000 7f613e1b7700 1 -- 172.21.15.46:0/1920007829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613810f340 con 0x7f6138101080 2023-06-23T01:01:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.864+0000 7f611ffff700 1 -- 172.21.15.46:0/1920007829 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f612801ac70 con 0x7f6138101080 2023-06-23T01:01:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.864+0000 7f611ffff700 1 -- 172.21.15.46:0/1920007829 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6128022ab0 con 0x7f6138101080 2023-06-23T01:01:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.864+0000 7f613e1b7700 1 -- 172.21.15.46:0/1920007829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613810f750 con 0x7f6138101080 2023-06-23T01:01:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.865+0000 7f611ffff700 1 -- 172.21.15.46:0/1920007829 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 43232+0+0 (secure 0 0 0) 0x7f6128022cf0 con 0x7f6138101080 2023-06-23T01:01:59.951 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.865+0000 7f613e1b7700 1 -- 172.21.15.46:0/1920007829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613804b970 con 0x7f6138101080 2023-06-23T01:01:59.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.865+0000 7f611ffff700 1 --2- 172.21.15.46:0/1920007829 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6120038800 0x7f612003acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:01:59.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.865+0000 7f611ffff700 1 -- 172.21.15.46:0/1920007829 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6128049dd0 con 0x7f6138101080 2023-06-23T01:01:59.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.865+0000 7f6136ffd700 1 --2- 172.21.15.46:0/1920007829 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6120038800 0x7f612003acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:01:59.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.866+0000 7f6136ffd700 1 --2- 172.21.15.46:0/1920007829 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6120038800 0x7f612003acb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f612c009990 tx=0x7f612c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:01:59.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:58.870+0000 7f611ffff700 1 -- 172.21.15.46:0/1920007829 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f6128010560 con 0x7f6138101080 2023-06-23T01:01:59.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.084+0000 7f613e1b7700 1 -- 172.21.15.46:0/1920007829 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6138102f20 con 0x7f6120038800 2023-06-23T01:01:59.952 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.331+0000 7f611ffff700 1 -- 172.21.15.46:0/1920007829 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f6138102f20 con 0x7f6120038800 2023-06-23T01:01:59.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.333+0000 7f613e1b7700 1 -- 172.21.15.46:0/1920007829 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6120038800 msgr2=0x7f612003acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:59.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.333+0000 7f613e1b7700 1 --2- 172.21.15.46:0/1920007829 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6120038800 0x7f612003acb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f612c009990 tx=0x7f612c006e40 comp rx=0 tx=0).stop 2023-06-23T01:01:59.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.333+0000 7f613e1b7700 1 -- 172.21.15.46:0/1920007829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 msgr2=0x7f613810ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:01:59.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.333+0000 7f613e1b7700 1 --2- 172.21.15.46:0/1920007829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f613810ec00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6128011650 tx=0x7f6128009250 comp rx=0 tx=0).stop 2023-06-23T01:01:59.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.333+0000 7f613e1b7700 1 -- 172.21.15.46:0/1920007829 shutdown_connections 2023-06-23T01:01:59.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.333+0000 7f613e1b7700 1 --2- 172.21.15.46:0/1920007829 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6120038800 0x7f612003acb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:59.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.333+0000 7f613e1b7700 1 --2- 172.21.15.46:0/1920007829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6138101080 0x7f613810ec00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:01:59.953 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.333+0000 7f613e1b7700 1 -- 172.21.15.46:0/1920007829 >> 172.21.15.46:0/1920007829 conn(0x7f61380fc800 msgr2=0x7f61380feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:01:59.954 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.333+0000 7f613e1b7700 1 -- 172.21.15.46:0/1920007829 shutdown_connections 2023-06-23T01:01:59.954 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:01:59.333+0000 7f613e1b7700 1 -- 172.21.15.46:0/1920007829 wait complete. 2023-06-23T01:01:59.954 INFO:teuthology.orchestra.run.smithi046.stdout:Fetching dashboard port number... 2023-06-23T01:01:59.987 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:59 smithi046 ceph-mon[84949]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:01:59.987 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:01:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:01.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:00 smithi046 ceph-mon[84949]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:01.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:00 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/1514482102' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-06-23T01:02:01.127 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 8443 2023-06-23T01:02:01.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.469+0000 7f6779433700 1 Processor -- start 2023-06-23T01:02:01.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.469+0000 7f6779433700 1 -- start start 2023-06-23T01:02:01.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.470+0000 7f6779433700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774076160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:01.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.470+0000 7f6779433700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6774078070 con 0x7f6774077c60 2023-06-23T01:02:01.128 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.470+0000 7f6772ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774076160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:01.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.470+0000 7f6772ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774076160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37684/0 (socket says 172.21.15.46:37684) 2023-06-23T01:02:01.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.470+0000 7f6772ffd700 1 -- 172.21.15.46:0/4258218343 learned_addr learned my addr 172.21.15.46:0/4258218343 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:01.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.470+0000 7f6772ffd700 1 -- 172.21.15.46:0/4258218343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67740766a0 con 0x7f6774077c60 2023-06-23T01:02:01.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.470+0000 7f6772ffd700 1 --2- 172.21.15.46:0/4258218343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774076160 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67640098e0 tx=0x7f6764009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4dc3d0edb0eaed server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:01.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.470+0000 7f6771ffb700 1 -- 172.21.15.46:0/4258218343 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6764009e80 con 0x7f6774077c60 2023-06-23T01:02:01.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.471+0000 7f6771ffb700 1 -- 172.21.15.46:0/4258218343 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f676400bd00 con 0x7f6774077c60 2023-06-23T01:02:01.129 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.471+0000 7f6771ffb700 1 -- 172.21.15.46:0/4258218343 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6764011c00 con 0x7f6774077c60 2023-06-23T01:02:01.130 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.471+0000 7f6779433700 1 -- 172.21.15.46:0/4258218343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 msgr2=0x7f6774076160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:01.130 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.471+0000 7f6779433700 1 --2- 172.21.15.46:0/4258218343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774076160 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67640098e0 tx=0x7f6764009250 comp rx=0 tx=0).stop 2023-06-23T01:02:01.130 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.471+0000 7f6779433700 1 -- 172.21.15.46:0/4258218343 shutdown_connections 2023-06-23T01:02:01.130 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.471+0000 7f6779433700 1 --2- 172.21.15.46:0/4258218343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774076160 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:01.130 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.471+0000 7f6779433700 1 -- 172.21.15.46:0/4258218343 >> 172.21.15.46:0/4258218343 conn(0x7f67740fd900 msgr2=0x7f67740ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:01.130 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.471+0000 7f6779433700 1 -- 172.21.15.46:0/4258218343 shutdown_connections 2023-06-23T01:02:01.131 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.471+0000 7f6779433700 1 -- 172.21.15.46:0/4258218343 wait complete. 2023-06-23T01:02:01.131 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.472+0000 7f6779433700 1 Processor -- start 2023-06-23T01:02:01.131 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.472+0000 7f6779433700 1 -- start start 2023-06-23T01:02:01.131 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.472+0000 7f6779433700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774112f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:01.131 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.472+0000 7f6779433700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6774113470 con 0x7f6774077c60 2023-06-23T01:02:01.131 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.472+0000 7f6772ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774112f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:01.132 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.472+0000 7f6772ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774112f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37692/0 (socket says 172.21.15.46:37692) 2023-06-23T01:02:01.132 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.472+0000 7f6772ffd700 1 -- 172.21.15.46:0/1514482102 learned_addr learned my addr 172.21.15.46:0/1514482102 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:01.132 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.473+0000 7f6772ffd700 1 -- 172.21.15.46:0/1514482102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67640094f0 con 0x7f6774077c60 2023-06-23T01:02:01.132 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.473+0000 7f6772ffd700 1 --2- 172.21.15.46:0/1514482102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774112f30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6764006fd0 tx=0x7f6764009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:01.132 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.473+0000 7f675b7fe700 1 -- 172.21.15.46:0/1514482102 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f676401a670 con 0x7f6774077c60 2023-06-23T01:02:01.132 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.473+0000 7f675b7fe700 1 -- 172.21.15.46:0/1514482102 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f676401ac70 con 0x7f6774077c60 2023-06-23T01:02:01.132 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.473+0000 7f6779433700 1 -- 172.21.15.46:0/1514482102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6774113670 con 0x7f6774077c60 2023-06-23T01:02:01.132 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.473+0000 7f675b7fe700 1 -- 172.21.15.46:0/1514482102 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6764022aa0 con 0x7f6774077c60 2023-06-23T01:02:01.133 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.473+0000 7f6779433700 1 -- 172.21.15.46:0/1514482102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6774114310 con 0x7f6774077c60 2023-06-23T01:02:01.133 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.474+0000 7f675b7fe700 1 -- 172.21.15.46:0/1514482102 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 43232+0+0 (secure 0 0 0) 0x7f6764022d10 con 0x7f6774077c60 2023-06-23T01:02:01.133 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.474+0000 7f675b7fe700 1 --2- 172.21.15.46:0/1514482102 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f675c036610 0x7f675c038ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:01.133 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.474+0000 7f6779433700 1 -- 172.21.15.46:0/1514482102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677405f060 con 0x7f6774077c60 2023-06-23T01:02:01.133 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.474+0000 7f675b7fe700 1 -- 172.21.15.46:0/1514482102 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6764049b80 con 0x7f6774077c60 2023-06-23T01:02:01.133 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.474+0000 7f67727fc700 1 --2- 172.21.15.46:0/1514482102 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f675c036610 0x7f675c038ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:01.134 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.474+0000 7f67727fc700 1 --2- 172.21.15.46:0/1514482102 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f675c036610 0x7f675c038ac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6768009990 tx=0x7f6768006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:01.134 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.479+0000 7f675b7fe700 1 -- 172.21.15.46:0/1514482102 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f6764010bb0 con 0x7f6774077c60 2023-06-23T01:02:01.134 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.672+0000 7f6779433700 1 -- 172.21.15.46:0/1514482102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f677404b970 con 0x7f6774077c60 2023-06-23T01:02:01.134 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.673+0000 7f675b7fe700 1 -- 172.21.15.46:0/1514482102 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f676401e030 con 0x7f6774077c60 2023-06-23T01:02:01.134 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.675+0000 7f6779433700 1 -- 172.21.15.46:0/1514482102 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f675c036610 msgr2=0x7f675c038ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:01.134 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.675+0000 7f6779433700 1 --2- 172.21.15.46:0/1514482102 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f675c036610 0x7f675c038ac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6768009990 tx=0x7f6768006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:01.134 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.675+0000 7f6779433700 1 -- 172.21.15.46:0/1514482102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 msgr2=0x7f6774112f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:01.135 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.675+0000 7f6779433700 1 --2- 172.21.15.46:0/1514482102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774112f30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6764006fd0 tx=0x7f6764009250 comp rx=0 tx=0).stop 2023-06-23T01:02:01.135 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.675+0000 7f6779433700 1 -- 172.21.15.46:0/1514482102 shutdown_connections 2023-06-23T01:02:01.135 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.675+0000 7f6779433700 1 --2- 172.21.15.46:0/1514482102 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f675c036610 0x7f675c038ac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:01.135 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.675+0000 7f6779433700 1 --2- 172.21.15.46:0/1514482102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774077c60 0x7f6774112f30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:01.135 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.675+0000 7f6779433700 1 -- 172.21.15.46:0/1514482102 >> 172.21.15.46:0/1514482102 conn(0x7f67740fd900 msgr2=0x7f67740fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:01.135 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.675+0000 7f6779433700 1 -- 172.21.15.46:0/1514482102 shutdown_connections 2023-06-23T01:02:01.135 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:00.675+0000 7f6779433700 1 -- 172.21.15.46:0/1514482102 wait complete. 2023-06-23T01:02:01.136 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-06-23T01:02:01.136 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2023-06-23T01:02:01.144 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-06-23T01:02:01.144 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2023-06-23T01:02:01.144 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2023-06-23T01:02:01.145 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-06-23T01:02:01.146 INFO:teuthology.orchestra.run.smithi046.stdout:Ceph Dashboard is now available at: 2023-06-23T01:02:01.146 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:01.146 INFO:teuthology.orchestra.run.smithi046.stdout: URL: https://smithi046.front.sepia.ceph.com:8443/ 2023-06-23T01:02:01.146 INFO:teuthology.orchestra.run.smithi046.stdout: User: admin 2023-06-23T01:02:01.147 INFO:teuthology.orchestra.run.smithi046.stdout: Password: wifqk5l3ws 2023-06-23T01:02:01.147 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:01.147 INFO:teuthology.orchestra.run.smithi046.stdout:Saving cluster configuration to /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config directory 2023-06-23T01:02:01.148 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling autotune for osd_memory_target 2023-06-23T01:02:02.190 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.579+0000 7fcd9ade7700 1 Processor -- start 2023-06-23T01:02:02.190 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.579+0000 7fcd9ade7700 1 -- start start 2023-06-23T01:02:02.191 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.579+0000 7fcd9ade7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:02.191 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.579+0000 7fcd9ade7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd941040f0 con 0x7fcd941037a0 2023-06-23T01:02:02.191 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.579+0000 7fcd98b83700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:02.191 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.579+0000 7fcd98b83700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37698/0 (socket says 172.21.15.46:37698) 2023-06-23T01:02:02.191 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.579+0000 7fcd98b83700 1 -- 172.21.15.46:0/169964280 learned_addr learned my addr 172.21.15.46:0/169964280 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:02.191 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.580+0000 7fcd98b83700 1 -- 172.21.15.46:0/169964280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd94104230 con 0x7fcd941037a0 2023-06-23T01:02:02.191 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.580+0000 7fcd98b83700 1 --2- 172.21.15.46:0/169964280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103bb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcd880098e0 tx=0x7fcd88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6298ad20a0bdead server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:02.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.580+0000 7fcd937fe700 1 -- 172.21.15.46:0/169964280 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd88009e80 con 0x7fcd941037a0 2023-06-23T01:02:02.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.580+0000 7fcd937fe700 1 -- 172.21.15.46:0/169964280 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcd8800bd00 con 0x7fcd941037a0 2023-06-23T01:02:02.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.580+0000 7fcd937fe700 1 -- 172.21.15.46:0/169964280 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd88011c00 con 0x7fcd941037a0 2023-06-23T01:02:02.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.580+0000 7fcd9ade7700 1 -- 172.21.15.46:0/169964280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 msgr2=0x7fcd94103bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:02.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.581+0000 7fcd9ade7700 1 --2- 172.21.15.46:0/169964280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103bb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcd880098e0 tx=0x7fcd88009250 comp rx=0 tx=0).stop 2023-06-23T01:02:02.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.581+0000 7fcd9ade7700 1 -- 172.21.15.46:0/169964280 shutdown_connections 2023-06-23T01:02:02.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.581+0000 7fcd9ade7700 1 --2- 172.21.15.46:0/169964280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103bb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:02.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.581+0000 7fcd9ade7700 1 -- 172.21.15.46:0/169964280 >> 172.21.15.46:0/169964280 conn(0x7fcd940fbf90 msgr2=0x7fcd940fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:02.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.581+0000 7fcd9ade7700 1 -- 172.21.15.46:0/169964280 shutdown_connections 2023-06-23T01:02:02.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.581+0000 7fcd9ade7700 1 -- 172.21.15.46:0/169964280 wait complete. 2023-06-23T01:02:02.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.581+0000 7fcd9ade7700 1 Processor -- start 2023-06-23T01:02:02.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.581+0000 7fcd9ade7700 1 -- start start 2023-06-23T01:02:02.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd9ade7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:02.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd9ade7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd941014f0 con 0x7fcd941037a0 2023-06-23T01:02:02.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd98b83700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:02.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd98b83700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37710/0 (socket says 172.21.15.46:37710) 2023-06-23T01:02:02.196 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd98b83700 1 -- 172.21.15.46:0/256502475 learned_addr learned my addr 172.21.15.46:0/256502475 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:02.197 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd98b83700 1 -- 172.21.15.46:0/256502475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd880094f0 con 0x7fcd941037a0 2023-06-23T01:02:02.197 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd98b83700 1 --2- 172.21.15.46:0/256502475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103070 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcd88006fd0 tx=0x7fcd88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:02.197 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd917fa700 1 -- 172.21.15.46:0/256502475 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd8801a670 con 0x7fcd941037a0 2023-06-23T01:02:02.197 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd917fa700 1 -- 172.21.15.46:0/256502475 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcd8801ac70 con 0x7fcd941037a0 2023-06-23T01:02:02.198 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd9ade7700 1 -- 172.21.15.46:0/256502475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd941016f0 con 0x7fcd941037a0 2023-06-23T01:02:02.198 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd917fa700 1 -- 172.21.15.46:0/256502475 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd88022a20 con 0x7fcd941037a0 2023-06-23T01:02:02.198 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.582+0000 7fcd9ade7700 1 -- 172.21.15.46:0/256502475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd94101b80 con 0x7fcd941037a0 2023-06-23T01:02:02.198 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.583+0000 7fcd917fa700 1 -- 172.21.15.46:0/256502475 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 43232+0+0 (secure 0 0 0) 0x7fcd880223c0 con 0x7fcd941037a0 2023-06-23T01:02:02.198 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.583+0000 7fcd917fa700 1 --2- 172.21.15.46:0/256502475 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcd84036660 0x7fcd84038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:02.198 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.583+0000 7fcd9ade7700 1 -- 172.21.15.46:0/256502475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9404b970 con 0x7fcd941037a0 2023-06-23T01:02:02.198 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.583+0000 7fcd917fa700 1 -- 172.21.15.46:0/256502475 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcd88049b20 con 0x7fcd941037a0 2023-06-23T01:02:02.199 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.584+0000 7fcd93fff700 1 --2- 172.21.15.46:0/256502475 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcd84036660 0x7fcd84038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:02.199 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.584+0000 7fcd93fff700 1 --2- 172.21.15.46:0/256502475 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcd84036660 0x7fcd84038b10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcd80009990 tx=0x7fcd80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:02.199 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.588+0000 7fcd917fa700 1 -- 172.21.15.46:0/256502475 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fcd8801e020 con 0x7fcd941037a0 2023-06-23T01:02:02.199 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.779+0000 7fcd9ade7700 1 -- 172.21.15.46:0/256502475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fcd94100ed0 con 0x7fcd941037a0 2023-06-23T01:02:02.199 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.780+0000 7fcd917fa700 1 -- 172.21.15.46:0/256502475 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fcd88022c10 con 0x7fcd941037a0 2023-06-23T01:02:02.199 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.782+0000 7fcd9ade7700 1 -- 172.21.15.46:0/256502475 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcd84036660 msgr2=0x7fcd84038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:02.199 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.782+0000 7fcd9ade7700 1 --2- 172.21.15.46:0/256502475 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcd84036660 0x7fcd84038b10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcd80009990 tx=0x7fcd80006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:02.200 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.782+0000 7fcd9ade7700 1 -- 172.21.15.46:0/256502475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 msgr2=0x7fcd94103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:02.200 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.782+0000 7fcd9ade7700 1 --2- 172.21.15.46:0/256502475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103070 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcd88006fd0 tx=0x7fcd88009250 comp rx=0 tx=0).stop 2023-06-23T01:02:02.200 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.782+0000 7fcd9ade7700 1 -- 172.21.15.46:0/256502475 shutdown_connections 2023-06-23T01:02:02.200 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.782+0000 7fcd9ade7700 1 --2- 172.21.15.46:0/256502475 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcd84036660 0x7fcd84038b10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:02.200 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.782+0000 7fcd9ade7700 1 --2- 172.21.15.46:0/256502475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd941037a0 0x7fcd94103070 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:02.200 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.782+0000 7fcd9ade7700 1 -- 172.21.15.46:0/256502475 >> 172.21.15.46:0/256502475 conn(0x7fcd940fbf90 msgr2=0x7fcd940fcc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:02.200 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.782+0000 7fcd9ade7700 1 -- 172.21.15.46:0/256502475 shutdown_connections 2023-06-23T01:02:02.200 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:01.782+0000 7fcd9ade7700 1 -- 172.21.15.46:0/256502475 wait complete. 2023-06-23T01:02:03.220 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:02 smithi046 ceph-mon[84949]: mgrmap e13: x(active, since 6s) 2023-06-23T01:02:03.230 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.581+0000 7fea9d6ef700 1 Processor -- start 2023-06-23T01:02:03.231 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.581+0000 7fea9d6ef700 1 -- start start 2023-06-23T01:02:03.231 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.582+0000 7fea9d6ef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98104dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:03.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.582+0000 7fea9d6ef700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98105310 con 0x7fea98070d60 2023-06-23T01:02:03.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.582+0000 7fea96ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98104dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:03.232 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.582+0000 7fea96ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98104dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37714/0 (socket says 172.21.15.46:37714) 2023-06-23T01:02:03.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.582+0000 7fea96ffd700 1 -- 172.21.15.46:0/2917965875 learned_addr learned my addr 172.21.15.46:0/2917965875 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:03.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.582+0000 7fea96ffd700 1 -- 172.21.15.46:0/2917965875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea98105450 con 0x7fea98070d60 2023-06-23T01:02:03.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.582+0000 7fea96ffd700 1 --2- 172.21.15.46:0/2917965875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98104dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fea800098e0 tx=0x7fea80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2bf089b6ca3da95 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:03.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.582+0000 7fea95ffb700 1 -- 172.21.15.46:0/2917965875 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea80009e80 con 0x7fea98070d60 2023-06-23T01:02:03.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.582+0000 7fea95ffb700 1 -- 172.21.15.46:0/2917965875 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fea8000bd00 con 0x7fea98070d60 2023-06-23T01:02:03.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.582+0000 7fea95ffb700 1 -- 172.21.15.46:0/2917965875 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea80011c00 con 0x7fea98070d60 2023-06-23T01:02:03.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.583+0000 7fea9d6ef700 1 -- 172.21.15.46:0/2917965875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 msgr2=0x7fea98104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:03.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.583+0000 7fea9d6ef700 1 --2- 172.21.15.46:0/2917965875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98104dd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fea800098e0 tx=0x7fea80009250 comp rx=0 tx=0).stop 2023-06-23T01:02:03.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.583+0000 7fea9d6ef700 1 -- 172.21.15.46:0/2917965875 shutdown_connections 2023-06-23T01:02:03.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.583+0000 7fea9d6ef700 1 --2- 172.21.15.46:0/2917965875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98104dd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:03.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.583+0000 7fea9d6ef700 1 -- 172.21.15.46:0/2917965875 >> 172.21.15.46:0/2917965875 conn(0x7fea980fc7e0 msgr2=0x7fea980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:03.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.583+0000 7fea9d6ef700 1 -- 172.21.15.46:0/2917965875 shutdown_connections 2023-06-23T01:02:03.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.583+0000 7fea9d6ef700 1 -- 172.21.15.46:0/2917965875 wait complete. 2023-06-23T01:02:03.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.584+0000 7fea9d6ef700 1 Processor -- start 2023-06-23T01:02:03.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.584+0000 7fea9d6ef700 1 -- start start 2023-06-23T01:02:03.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.584+0000 7fea9d6ef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98102850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:03.237 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.584+0000 7fea9d6ef700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98100cd0 con 0x7fea98070d60 2023-06-23T01:02:03.237 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.584+0000 7fea96ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98102850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:03.237 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.584+0000 7fea96ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98102850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37724/0 (socket says 172.21.15.46:37724) 2023-06-23T01:02:03.237 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.584+0000 7fea96ffd700 1 -- 172.21.15.46:0/1630070500 learned_addr learned my addr 172.21.15.46:0/1630070500 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:03.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.584+0000 7fea96ffd700 1 -- 172.21.15.46:0/1630070500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea800094f0 con 0x7fea98070d60 2023-06-23T01:02:03.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.584+0000 7fea96ffd700 1 --2- 172.21.15.46:0/1630070500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98102850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fea80009fd0 tx=0x7fea80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:03.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.585+0000 7fea8f7fe700 1 -- 172.21.15.46:0/1630070500 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea8001a670 con 0x7fea98070d60 2023-06-23T01:02:03.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.585+0000 7fea8f7fe700 1 -- 172.21.15.46:0/1630070500 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fea8001ac70 con 0x7fea98070d60 2023-06-23T01:02:03.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.585+0000 7fea8f7fe700 1 -- 172.21.15.46:0/1630070500 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea80022bb0 con 0x7fea98070d60 2023-06-23T01:02:03.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.585+0000 7fea9d6ef700 1 -- 172.21.15.46:0/1630070500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea98100ed0 con 0x7fea98070d60 2023-06-23T01:02:03.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.585+0000 7fea9d6ef700 1 -- 172.21.15.46:0/1630070500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea98101360 con 0x7fea98070d60 2023-06-23T01:02:03.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.585+0000 7fea8f7fe700 1 -- 172.21.15.46:0/1630070500 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fea8001a7d0 con 0x7fea98070d60 2023-06-23T01:02:03.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.585+0000 7fea9d6ef700 1 -- 172.21.15.46:0/1630070500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea9804b970 con 0x7fea98070d60 2023-06-23T01:02:03.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.586+0000 7fea8f7fe700 1 --2- 172.21.15.46:0/1630070500 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fea840366b0 0x7fea84038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:03.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.586+0000 7fea8f7fe700 1 -- 172.21.15.46:0/1630070500 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fea8004a760 con 0x7fea98070d60 2023-06-23T01:02:03.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.586+0000 7fea967fc700 1 --2- 172.21.15.46:0/1630070500 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fea840366b0 0x7fea84038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:03.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.586+0000 7fea967fc700 1 --2- 172.21.15.46:0/1630070500 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fea840366b0 0x7fea84038b60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fea88009990 tx=0x7fea88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:03.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.591+0000 7fea8f7fe700 1 -- 172.21.15.46:0/1630070500 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fea8002ab80 con 0x7fea98070d60 2023-06-23T01:02:03.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.891+0000 7fea9d6ef700 1 -- 172.21.15.46:0/1630070500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fea981017e0 con 0x7fea98070d60 2023-06-23T01:02:03.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.893+0000 7fea8f7fe700 1 -- 172.21.15.46:0/1630070500 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fea8002ab80 con 0x7fea98070d60 2023-06-23T01:02:03.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.895+0000 7fea9d6ef700 1 -- 172.21.15.46:0/1630070500 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fea840366b0 msgr2=0x7fea84038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:03.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.895+0000 7fea9d6ef700 1 --2- 172.21.15.46:0/1630070500 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fea840366b0 0x7fea84038b60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fea88009990 tx=0x7fea88006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:03.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.895+0000 7fea9d6ef700 1 -- 172.21.15.46:0/1630070500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 msgr2=0x7fea98102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:03.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.895+0000 7fea9d6ef700 1 --2- 172.21.15.46:0/1630070500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98102850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fea80009fd0 tx=0x7fea80009250 comp rx=0 tx=0).stop 2023-06-23T01:02:03.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.895+0000 7fea9d6ef700 1 -- 172.21.15.46:0/1630070500 shutdown_connections 2023-06-23T01:02:03.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.895+0000 7fea9d6ef700 1 --2- 172.21.15.46:0/1630070500 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fea840366b0 0x7fea84038b60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:03.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.895+0000 7fea9d6ef700 1 --2- 172.21.15.46:0/1630070500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea98070d60 0x7fea98102850 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:03.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.895+0000 7fea9d6ef700 1 -- 172.21.15.46:0/1630070500 >> 172.21.15.46:0/1630070500 conn(0x7fea980fc7e0 msgr2=0x7fea980fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:03.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.895+0000 7fea9d6ef700 1 -- 172.21.15.46:0/1630070500 shutdown_connections 2023-06-23T01:02:03.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2023-06-23T01:02:02.895+0000 7fea9d6ef700 1 -- 172.21.15.46:0/1630070500 wait complete. 2023-06-23T01:02:03.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-06-23T01:02:03.245 INFO:teuthology.orchestra.run.smithi046.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-06-23T01:02:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-06-23T01:02:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:03.246 INFO:teuthology.orchestra.run.smithi046.stdout:Or, if you are only running a single cluster on this host: 2023-06-23T01:02:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-06-23T01:02:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:03.247 INFO:teuthology.orchestra.run.smithi046.stdout:Please consider enabling telemetry to help improve Ceph: 2023-06-23T01:02:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: ceph telemetry on 2023-06-23T01:02:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:03.247 INFO:teuthology.orchestra.run.smithi046.stdout:For more information see: 2023-06-23T01:02:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2023-06-23T01:02:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:03.248 INFO:teuthology.orchestra.run.smithi046.stdout:Bootstrap complete. 2023-06-23T01:02:03.264 INFO:tasks.cephadm:Fetching config... 2023-06-23T01:02:03.264 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T01:02:03.265 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-06-23T01:02:03.280 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-06-23T01:02:03.281 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T01:02:03.281 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-06-23T01:02:03.337 INFO:tasks.cephadm:Fetching mon keyring... 2023-06-23T01:02:03.337 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T01:02:03.337 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-06-23T01:02:03.404 INFO:tasks.cephadm:Fetching pub ssh key... 2023-06-23T01:02:03.404 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T01:02:03.405 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-06-23T01:02:03.460 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-06-23T01:02:03.461 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwfVdaEdjxWRLztCaJ41kP7iCLhF9/XDXS4eLspHS4XaQr02n4Ko2a6LeYOwLTSrUD6jt6db+IXUI7Sv8zm4j9CGyJbXuMekJbqNB0GZscdv/9jm7E3ly3KYn2FV//DBn8wC+DUliZLciUmVg4VTAaIW4UO0OLEYlm+ZDz8DRiA+LZ6/gbEQBF6cL4xz/hsAx2dTn2kpkPM13q/O6L2cfzhVkt+IMJspFbmSuxIkWRL4W+UFbZbXo4bjVagNDs3pNudF5FkjAqYLW8qrL+OkOtrsPDakeqHBRrDRviRpYZlQnRMV72nKO5ROYxSuyyGmMISQ8QZkdIcw/UnpPcnQWmiuwAcY4OfL8xrEQUq+0ttmjWrw0IRirYbnaebUZrQ/q3A6EA3t2a3S2RsGvXIRUEzWiPmfE+At6WhJB33vTliOeMDmNw9a/6e9JTMjvmqOrI62FTQGABc5pyeeS9Gclx4QHD4pZPYEchnOWLJPi7uPiLSiD2oISDxENwwdb4wAk= ceph-458642ee-1161-11ee-9b2b-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-06-23T01:02:03.549 INFO:teuthology.orchestra.run.smithi046.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwfVdaEdjxWRLztCaJ41kP7iCLhF9/XDXS4eLspHS4XaQr02n4Ko2a6LeYOwLTSrUD6jt6db+IXUI7Sv8zm4j9CGyJbXuMekJbqNB0GZscdv/9jm7E3ly3KYn2FV//DBn8wC+DUliZLciUmVg4VTAaIW4UO0OLEYlm+ZDz8DRiA+LZ6/gbEQBF6cL4xz/hsAx2dTn2kpkPM13q/O6L2cfzhVkt+IMJspFbmSuxIkWRL4W+UFbZbXo4bjVagNDs3pNudF5FkjAqYLW8qrL+OkOtrsPDakeqHBRrDRviRpYZlQnRMV72nKO5ROYxSuyyGmMISQ8QZkdIcw/UnpPcnQWmiuwAcY4OfL8xrEQUq+0ttmjWrw0IRirYbnaebUZrQ/q3A6EA3t2a3S2RsGvXIRUEzWiPmfE+At6WhJB33vTliOeMDmNw9a/6e9JTMjvmqOrI62FTQGABc5pyeeS9Gclx4QHD4pZPYEchnOWLJPi7uPiLSiD2oISDxENwwdb4wAk= ceph-458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:02:03.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwfVdaEdjxWRLztCaJ41kP7iCLhF9/XDXS4eLspHS4XaQr02n4Ko2a6LeYOwLTSrUD6jt6db+IXUI7Sv8zm4j9CGyJbXuMekJbqNB0GZscdv/9jm7E3ly3KYn2FV//DBn8wC+DUliZLciUmVg4VTAaIW4UO0OLEYlm+ZDz8DRiA+LZ6/gbEQBF6cL4xz/hsAx2dTn2kpkPM13q/O6L2cfzhVkt+IMJspFbmSuxIkWRL4W+UFbZbXo4bjVagNDs3pNudF5FkjAqYLW8qrL+OkOtrsPDakeqHBRrDRviRpYZlQnRMV72nKO5ROYxSuyyGmMISQ8QZkdIcw/UnpPcnQWmiuwAcY4OfL8xrEQUq+0ttmjWrw0IRirYbnaebUZrQ/q3A6EA3t2a3S2RsGvXIRUEzWiPmfE+At6WhJB33vTliOeMDmNw9a/6e9JTMjvmqOrI62FTQGABc5pyeeS9Gclx4QHD4pZPYEchnOWLJPi7uPiLSiD2oISDxENwwdb4wAk= ceph-458642ee-1161-11ee-9b2b-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-06-23T01:02:03.602 INFO:teuthology.orchestra.run.smithi092.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwfVdaEdjxWRLztCaJ41kP7iCLhF9/XDXS4eLspHS4XaQr02n4Ko2a6LeYOwLTSrUD6jt6db+IXUI7Sv8zm4j9CGyJbXuMekJbqNB0GZscdv/9jm7E3ly3KYn2FV//DBn8wC+DUliZLciUmVg4VTAaIW4UO0OLEYlm+ZDz8DRiA+LZ6/gbEQBF6cL4xz/hsAx2dTn2kpkPM13q/O6L2cfzhVkt+IMJspFbmSuxIkWRL4W+UFbZbXo4bjVagNDs3pNudF5FkjAqYLW8qrL+OkOtrsPDakeqHBRrDRviRpYZlQnRMV72nKO5ROYxSuyyGmMISQ8QZkdIcw/UnpPcnQWmiuwAcY4OfL8xrEQUq+0ttmjWrw0IRirYbnaebUZrQ/q3A6EA3t2a3S2RsGvXIRUEzWiPmfE+At6WhJB33vTliOeMDmNw9a/6e9JTMjvmqOrI62FTQGABc5pyeeS9Gclx4QHD4pZPYEchnOWLJPi7uPiLSiD2oISDxENwwdb4wAk= ceph-458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:02:03.614 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwfVdaEdjxWRLztCaJ41kP7iCLhF9/XDXS4eLspHS4XaQr02n4Ko2a6LeYOwLTSrUD6jt6db+IXUI7Sv8zm4j9CGyJbXuMekJbqNB0GZscdv/9jm7E3ly3KYn2FV//DBn8wC+DUliZLciUmVg4VTAaIW4UO0OLEYlm+ZDz8DRiA+LZ6/gbEQBF6cL4xz/hsAx2dTn2kpkPM13q/O6L2cfzhVkt+IMJspFbmSuxIkWRL4W+UFbZbXo4bjVagNDs3pNudF5FkjAqYLW8qrL+OkOtrsPDakeqHBRrDRviRpYZlQnRMV72nKO5ROYxSuyyGmMISQ8QZkdIcw/UnpPcnQWmiuwAcY4OfL8xrEQUq+0ttmjWrw0IRirYbnaebUZrQ/q3A6EA3t2a3S2RsGvXIRUEzWiPmfE+At6WhJB33vTliOeMDmNw9a/6e9JTMjvmqOrI62FTQGABc5pyeeS9Gclx4QHD4pZPYEchnOWLJPi7uPiLSiD2oISDxENwwdb4wAk= ceph-458642ee-1161-11ee-9b2b-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-06-23T01:02:03.658 INFO:teuthology.orchestra.run.smithi131.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwfVdaEdjxWRLztCaJ41kP7iCLhF9/XDXS4eLspHS4XaQr02n4Ko2a6LeYOwLTSrUD6jt6db+IXUI7Sv8zm4j9CGyJbXuMekJbqNB0GZscdv/9jm7E3ly3KYn2FV//DBn8wC+DUliZLciUmVg4VTAaIW4UO0OLEYlm+ZDz8DRiA+LZ6/gbEQBF6cL4xz/hsAx2dTn2kpkPM13q/O6L2cfzhVkt+IMJspFbmSuxIkWRL4W+UFbZbXo4bjVagNDs3pNudF5FkjAqYLW8qrL+OkOtrsPDakeqHBRrDRviRpYZlQnRMV72nKO5ROYxSuyyGmMISQ8QZkdIcw/UnpPcnQWmiuwAcY4OfL8xrEQUq+0ttmjWrw0IRirYbnaebUZrQ/q3A6EA3t2a3S2RsGvXIRUEzWiPmfE+At6WhJB33vTliOeMDmNw9a/6e9JTMjvmqOrI62FTQGABc5pyeeS9Gclx4QHD4pZPYEchnOWLJPi7uPiLSiD2oISDxENwwdb4wAk= ceph-458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:02:03.670 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-06-23T01:02:04.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:03 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/1630070500' entity='client.admin' 2023-06-23T01:02:04.522 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:02:06.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.081+0000 7fc27edea700 1 -- 172.21.15.46:0/274735295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2780fb550 msgr2=0x7fc2780fb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:06.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.081+0000 7fc27edea700 1 --2- 172.21.15.46:0/274735295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2780fb550 0x7fc2780fb960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc26c0048c0 tx=0x7fc26c009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:06.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.081+0000 7fc27edea700 1 -- 172.21.15.46:0/274735295 shutdown_connections 2023-06-23T01:02:06.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.081+0000 7fc27edea700 1 --2- 172.21.15.46:0/274735295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2780fb550 0x7fc2780fb960 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:06.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.081+0000 7fc27edea700 1 -- 172.21.15.46:0/274735295 >> 172.21.15.46:0/274735295 conn(0x7fc2780f6cb0 msgr2=0x7fc2780f90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:06.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.081+0000 7fc27edea700 1 -- 172.21.15.46:0/274735295 shutdown_connections 2023-06-23T01:02:06.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.081+0000 7fc27edea700 1 -- 172.21.15.46:0/274735295 wait complete. 2023-06-23T01:02:06.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.082+0000 7fc27edea700 1 Processor -- start 2023-06-23T01:02:06.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.082+0000 7fc27edea700 1 -- start start 2023-06-23T01:02:06.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.082+0000 7fc27edea700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2780fb550 0x7fc27810f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:06.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.082+0000 7fc27edea700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2780662f0 con 0x7fc2780fb550 2023-06-23T01:02:06.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.082+0000 7fc27cb86700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2780fb550 0x7fc27810f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:06.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.082+0000 7fc27cb86700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2780fb550 0x7fc27810f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59872/0 (socket says 172.21.15.46:59872) 2023-06-23T01:02:06.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.082+0000 7fc27cb86700 1 -- 172.21.15.46:0/3405075548 learned_addr learned my addr 172.21.15.46:0/3405075548 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:06.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.083+0000 7fc27cb86700 1 -- 172.21.15.46:0/3405075548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c005040 con 0x7fc2780fb550 2023-06-23T01:02:06.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.083+0000 7fc27cb86700 1 --2- 172.21.15.46:0/3405075548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2780fb550 0x7fc27810f3e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc26c000c00 tx=0x7fc26c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:06.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.083+0000 7fc2757fa700 1 -- 172.21.15.46:0/3405075548 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc26c00e040 con 0x7fc2780fb550 2023-06-23T01:02:06.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.083+0000 7fc2757fa700 1 -- 172.21.15.46:0/3405075548 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc26c013420 con 0x7fc2780fb550 2023-06-23T01:02:06.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.083+0000 7fc27edea700 1 -- 172.21.15.46:0/3405075548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2780664f0 con 0x7fc2780fb550 2023-06-23T01:02:06.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.083+0000 7fc2757fa700 1 -- 172.21.15.46:0/3405075548 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc26c01c3f0 con 0x7fc2780fb550 2023-06-23T01:02:06.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.083+0000 7fc27edea700 1 -- 172.21.15.46:0/3405075548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc27810fb60 con 0x7fc2780fb550 2023-06-23T01:02:06.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.084+0000 7fc2757fa700 1 -- 172.21.15.46:0/3405075548 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fc26c01c660 con 0x7fc2780fb550 2023-06-23T01:02:06.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.084+0000 7fc2757fa700 1 --2- 172.21.15.46:0/3405075548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc2680366b0 0x7fc268038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:06.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.084+0000 7fc2757fa700 1 -- 172.21.15.46:0/3405075548 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc26c049570 con 0x7fc2780fb550 2023-06-23T01:02:06.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.084+0000 7fc27edea700 1 -- 172.21.15.46:0/3405075548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc278105040 con 0x7fc2780fb550 2023-06-23T01:02:06.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.084+0000 7fc277fff700 1 --2- 172.21.15.46:0/3405075548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc2680366b0 0x7fc268038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:06.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.084+0000 7fc277fff700 1 --2- 172.21.15.46:0/3405075548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc2680366b0 0x7fc268038b60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc264009990 tx=0x7fc264006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:06.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.089+0000 7fc2757fa700 1 -- 172.21.15.46:0/3405075548 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fc26c025400 con 0x7fc2780fb550 2023-06-23T01:02:06.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.280+0000 7fc27edea700 1 -- 172.21.15.46:0/3405075548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fc27804b970 con 0x7fc2780fb550 2023-06-23T01:02:06.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.283+0000 7fc2757fa700 1 -- 172.21.15.46:0/3405075548 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc26c011b60 con 0x7fc2780fb550 2023-06-23T01:02:06.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.285+0000 7fc27edea700 1 -- 172.21.15.46:0/3405075548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc2680366b0 msgr2=0x7fc268038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:06.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.285+0000 7fc27edea700 1 --2- 172.21.15.46:0/3405075548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc2680366b0 0x7fc268038b60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc264009990 tx=0x7fc264006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:06.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.285+0000 7fc27edea700 1 -- 172.21.15.46:0/3405075548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2780fb550 msgr2=0x7fc27810f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:06.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.285+0000 7fc27edea700 1 --2- 172.21.15.46:0/3405075548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2780fb550 0x7fc27810f3e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc26c000c00 tx=0x7fc26c009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:06.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.286+0000 7fc27edea700 1 -- 172.21.15.46:0/3405075548 shutdown_connections 2023-06-23T01:02:06.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.286+0000 7fc27edea700 1 --2- 172.21.15.46:0/3405075548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc2680366b0 0x7fc268038b60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:06.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.286+0000 7fc27edea700 1 --2- 172.21.15.46:0/3405075548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2780fb550 0x7fc27810f3e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:06.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.286+0000 7fc27edea700 1 -- 172.21.15.46:0/3405075548 >> 172.21.15.46:0/3405075548 conn(0x7fc2780f6cb0 msgr2=0x7fc2780f7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:06.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.286+0000 7fc27edea700 1 -- 172.21.15.46:0/3405075548 shutdown_connections 2023-06-23T01:02:06.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:06.286+0000 7fc27edea700 1 -- 172.21.15.46:0/3405075548 wait complete. 2023-06-23T01:02:06.678 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-06-23T01:02:06.679 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-06-23T01:02:07.271 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:02:07.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:07.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:07.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi046", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:02:07.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:07.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:07.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:07.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:06 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3405075548' entity='client.admin' 2023-06-23T01:02:07.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:07.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:08.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.458+0000 7f231db31700 1 -- 172.21.15.46:0/3702326124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23180fdd60 msgr2=0x7f23180fe170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:08.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.458+0000 7f231db31700 1 --2- 172.21.15.46:0/3702326124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23180fdd60 0x7f23180fe170 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f23080048c0 tx=0x7f2308009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:08.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.458+0000 7f231db31700 1 -- 172.21.15.46:0/3702326124 shutdown_connections 2023-06-23T01:02:08.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.458+0000 7f231db31700 1 --2- 172.21.15.46:0/3702326124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23180fdd60 0x7f23180fe170 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:08.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.458+0000 7f231db31700 1 -- 172.21.15.46:0/3702326124 >> 172.21.15.46:0/3702326124 conn(0x7f23180f6460 msgr2=0x7f23180f8870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:08.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.458+0000 7f231db31700 1 -- 172.21.15.46:0/3702326124 shutdown_connections 2023-06-23T01:02:08.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.458+0000 7f231db31700 1 -- 172.21.15.46:0/3702326124 wait complete. 2023-06-23T01:02:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.458+0000 7f231db31700 1 Processor -- start 2023-06-23T01:02:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.458+0000 7f231db31700 1 -- start start 2023-06-23T01:02:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f231db31700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23180fdd60 0x7f231810c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f231db31700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231810ccd0 con 0x7f23180fdd60 2023-06-23T01:02:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f23177fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23180fdd60 0x7f231810c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f23177fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23180fdd60 0x7f231810c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59878/0 (socket says 172.21.15.46:59878) 2023-06-23T01:02:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f23177fe700 1 -- 172.21.15.46:0/3187832098 learned_addr learned my addr 172.21.15.46:0/3187832098 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f23177fe700 1 -- 172.21.15.46:0/3187832098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2308005040 con 0x7f23180fdd60 2023-06-23T01:02:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f23177fe700 1 --2- 172.21.15.46:0/3187832098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23180fdd60 0x7f231810c790 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2308008f00 tx=0x7f2308009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f22fffff700 1 -- 172.21.15.46:0/3187832098 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f230800e040 con 0x7f23180fdd60 2023-06-23T01:02:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f231db31700 1 -- 172.21.15.46:0/3187832098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231810ced0 con 0x7f23180fdd60 2023-06-23T01:02:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f22fffff700 1 -- 172.21.15.46:0/3187832098 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f230800be50 con 0x7f23180fdd60 2023-06-23T01:02:08.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f22fffff700 1 -- 172.21.15.46:0/3187832098 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2308013d70 con 0x7f23180fdd60 2023-06-23T01:02:08.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.459+0000 7f231db31700 1 -- 172.21.15.46:0/3187832098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231810db70 con 0x7f23180fdd60 2023-06-23T01:02:08.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.460+0000 7f22fffff700 1 -- 172.21.15.46:0/3187832098 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f2308025400 con 0x7f23180fdd60 2023-06-23T01:02:08.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.460+0000 7f22fffff700 1 --2- 172.21.15.46:0/3187832098 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f2300036660 0x7f2300038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:08.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.460+0000 7f22fffff700 1 -- 172.21.15.46:0/3187832098 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f23080495d0 con 0x7f23180fdd60 2023-06-23T01:02:08.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.460+0000 7f2316ffd700 1 --2- 172.21.15.46:0/3187832098 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f2300036660 0x7f2300038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:08.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.460+0000 7f231db31700 1 -- 172.21.15.46:0/3187832098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f231805f060 con 0x7f23180fdd60 2023-06-23T01:02:08.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.461+0000 7f2316ffd700 1 --2- 172.21.15.46:0/3187832098 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f2300036660 0x7f2300038b10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f230c009990 tx=0x7f230c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:08.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.465+0000 7f22fffff700 1 -- 172.21.15.46:0/3187832098 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f2308011960 con 0x7f23180fdd60 2023-06-23T01:02:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.663+0000 7f231db31700 1 -- 172.21.15.46:0/3187832098 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f231810ddb0 con 0x7f2300036660 2023-06-23T01:02:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.666+0000 7f22fffff700 1 -- 172.21.15.46:0/3187832098 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f231810ddb0 con 0x7f2300036660 2023-06-23T01:02:08.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.668+0000 7f231db31700 1 -- 172.21.15.46:0/3187832098 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f2300036660 msgr2=0x7f2300038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:08.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.668+0000 7f231db31700 1 --2- 172.21.15.46:0/3187832098 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f2300036660 0x7f2300038b10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f230c009990 tx=0x7f230c006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:08.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.668+0000 7f231db31700 1 -- 172.21.15.46:0/3187832098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23180fdd60 msgr2=0x7f231810c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:08.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.668+0000 7f231db31700 1 --2- 172.21.15.46:0/3187832098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23180fdd60 0x7f231810c790 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2308008f00 tx=0x7f2308009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:08.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.668+0000 7f231db31700 1 -- 172.21.15.46:0/3187832098 shutdown_connections 2023-06-23T01:02:08.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.668+0000 7f231db31700 1 --2- 172.21.15.46:0/3187832098 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f2300036660 0x7f2300038b10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:08.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.668+0000 7f231db31700 1 --2- 172.21.15.46:0/3187832098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23180fdd60 0x7f231810c790 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:08.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.668+0000 7f231db31700 1 -- 172.21.15.46:0/3187832098 >> 172.21.15.46:0/3187832098 conn(0x7f23180f6460 msgr2=0x7f23180f7290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:08.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.669+0000 7f231db31700 1 -- 172.21.15.46:0/3187832098 shutdown_connections 2023-06-23T01:02:08.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:08.669+0000 7f231db31700 1 -- 172.21.15.46:0/3187832098 wait complete. 2023-06-23T01:02:09.009 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi092 2023-06-23T01:02:09.010 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:02:09.010 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/etc/ceph/ceph.conf 2023-06-23T01:02:09.028 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:02:09.028 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-06-23T01:02:09.085 INFO:tasks.cephadm:Adding host smithi092 to orchestrator... 2023-06-23T01:02:09.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch host add smithi092 2023-06-23T01:02:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:02:09.936 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:09 smithi046 ceph-mon[84949]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:09.936 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:09 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:09.936 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:09 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:09.937 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:09 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:09.937 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:09 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:09.937 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:09 smithi046 ceph-mon[84949]: Updating smithi046:/etc/ceph/ceph.conf 2023-06-23T01:02:09.937 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:09 smithi046 ceph-mon[84949]: Updating smithi046:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:09.937 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:09 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:09.937 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:09 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:09.937 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:09 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:10.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.829+0000 7fc97d371700 1 -- 172.21.15.46:0/484129200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9781026c0 msgr2=0x7fc978102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:10.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.829+0000 7fc97d371700 1 --2- 172.21.15.46:0/484129200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9781026c0 0x7fc978102a90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc9680048c0 tx=0x7fc968009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:10.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.829+0000 7fc97d371700 1 -- 172.21.15.46:0/484129200 shutdown_connections 2023-06-23T01:02:10.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.829+0000 7fc97d371700 1 --2- 172.21.15.46:0/484129200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9781026c0 0x7fc978102a90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:10.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.829+0000 7fc97d371700 1 -- 172.21.15.46:0/484129200 >> 172.21.15.46:0/484129200 conn(0x7fc9780f81e0 msgr2=0x7fc9780fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:10.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.829+0000 7fc97d371700 1 -- 172.21.15.46:0/484129200 shutdown_connections 2023-06-23T01:02:10.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.829+0000 7fc97d371700 1 -- 172.21.15.46:0/484129200 wait complete. 2023-06-23T01:02:10.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.830+0000 7fc97d371700 1 Processor -- start 2023-06-23T01:02:10.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.830+0000 7fc97d371700 1 -- start start 2023-06-23T01:02:10.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.830+0000 7fc97d371700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9781026c0 0x7fc9781127f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:10.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.830+0000 7fc97d371700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc978112d30 con 0x7fc9781026c0 2023-06-23T01:02:10.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.830+0000 7fc976ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9781026c0 0x7fc9781127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:10.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.830+0000 7fc976ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9781026c0 0x7fc9781127f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59910/0 (socket says 172.21.15.46:59910) 2023-06-23T01:02:10.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.830+0000 7fc976ffd700 1 -- 172.21.15.46:0/792483930 learned_addr learned my addr 172.21.15.46:0/792483930 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:10.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.830+0000 7fc976ffd700 1 -- 172.21.15.46:0/792483930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc968005040 con 0x7fc9781026c0 2023-06-23T01:02:10.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.830+0000 7fc976ffd700 1 --2- 172.21.15.46:0/792483930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9781026c0 0x7fc9781127f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc968009aa0 tx=0x7fc968009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:10.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.831+0000 7fc96f7fe700 1 -- 172.21.15.46:0/792483930 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc96800e040 con 0x7fc9781026c0 2023-06-23T01:02:10.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.831+0000 7fc97d371700 1 -- 172.21.15.46:0/792483930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc978112f30 con 0x7fc9781026c0 2023-06-23T01:02:10.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.831+0000 7fc96f7fe700 1 -- 172.21.15.46:0/792483930 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc96800be00 con 0x7fc9781026c0 2023-06-23T01:02:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.831+0000 7fc96f7fe700 1 -- 172.21.15.46:0/792483930 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc968013d40 con 0x7fc9781026c0 2023-06-23T01:02:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.831+0000 7fc97d371700 1 -- 172.21.15.46:0/792483930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97810c6b0 con 0x7fc9781026c0 2023-06-23T01:02:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.832+0000 7fc96f7fe700 1 -- 172.21.15.46:0/792483930 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fc968025400 con 0x7fc9781026c0 2023-06-23T01:02:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.832+0000 7fc97d371700 1 -- 172.21.15.46:0/792483930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97805f060 con 0x7fc9781026c0 2023-06-23T01:02:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.832+0000 7fc96f7fe700 1 --2- 172.21.15.46:0/792483930 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc964036660 0x7fc964038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.832+0000 7fc96f7fe700 1 -- 172.21.15.46:0/792483930 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc968049710 con 0x7fc9781026c0 2023-06-23T01:02:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.832+0000 7fc9767fc700 1 --2- 172.21.15.46:0/792483930 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc964036660 0x7fc964038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:10.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.832+0000 7fc9767fc700 1 --2- 172.21.15.46:0/792483930 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc964036660 0x7fc964038b10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc960009990 tx=0x7fc960006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:10.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:10.837+0000 7fc96f7fe700 1 -- 172.21.15.46:0/792483930 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fc968011960 con 0x7fc9781026c0 2023-06-23T01:02:11.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:11.040+0000 7fc97d371700 1 -- 172.21.15.46:0/792483930 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi092", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9781124c0 con 0x7fc964036660 2023-06-23T01:02:11.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:10 smithi046 ceph-mon[84949]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2023-06-23T01:02:11.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:10 smithi046 ceph-mon[84949]: Updating smithi046:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.client.admin.keyring 2023-06-23T01:02:12.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:12 smithi046 ceph-mon[84949]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi092", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:12.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:12 smithi046 ceph-mon[84949]: Deploying cephadm binary to smithi092 2023-06-23T01:02:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:12.930+0000 7fc96f7fe700 1 -- 172.21.15.46:0/792483930 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fc9781124c0 con 0x7fc964036660 2023-06-23T01:02:12.935 INFO:teuthology.orchestra.run.smithi046.stdout:Added host 'smithi092' with addr '172.21.15.92' 2023-06-23T01:02:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:12.932+0000 7fc97d371700 1 -- 172.21.15.46:0/792483930 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc964036660 msgr2=0x7fc964038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:12.932+0000 7fc97d371700 1 --2- 172.21.15.46:0/792483930 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc964036660 0x7fc964038b10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc960009990 tx=0x7fc960006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:12.932+0000 7fc97d371700 1 -- 172.21.15.46:0/792483930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9781026c0 msgr2=0x7fc9781127f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:12.932+0000 7fc97d371700 1 --2- 172.21.15.46:0/792483930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9781026c0 0x7fc9781127f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc968009aa0 tx=0x7fc968009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:12.932+0000 7fc97d371700 1 -- 172.21.15.46:0/792483930 shutdown_connections 2023-06-23T01:02:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:12.932+0000 7fc97d371700 1 --2- 172.21.15.46:0/792483930 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc964036660 0x7fc964038b10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:12.932+0000 7fc97d371700 1 --2- 172.21.15.46:0/792483930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9781026c0 0x7fc9781127f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:12.932+0000 7fc97d371700 1 -- 172.21.15.46:0/792483930 >> 172.21.15.46:0/792483930 conn(0x7fc9780f81e0 msgr2=0x7fc9780fa0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:12.933+0000 7fc97d371700 1 -- 172.21.15.46:0/792483930 shutdown_connections 2023-06-23T01:02:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:12.933+0000 7fc97d371700 1 -- 172.21.15.46:0/792483930 wait complete. 2023-06-23T01:02:13.333 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch host ls --format=json 2023-06-23T01:02:13.929 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:02:14.258 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:13 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:14.258 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:13 smithi046 ceph-mon[84949]: Added host smithi092 2023-06-23T01:02:14.258 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:13 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:14.258 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:13 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:15.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.031+0000 7f63a7d14700 1 -- 172.21.15.46:0/3230143798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a00fb3b0 msgr2=0x7f63a00fb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:15.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.031+0000 7f63a7d14700 1 --2- 172.21.15.46:0/3230143798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a00fb3b0 0x7f63a00fb780 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6398009c60 tx=0x7f6398004d10 comp rx=0 tx=0).stop 2023-06-23T01:02:15.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.031+0000 7f63a7d14700 1 -- 172.21.15.46:0/3230143798 shutdown_connections 2023-06-23T01:02:15.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.031+0000 7f63a7d14700 1 --2- 172.21.15.46:0/3230143798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a00fb3b0 0x7f63a00fb780 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:15.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.031+0000 7f63a7d14700 1 -- 172.21.15.46:0/3230143798 >> 172.21.15.46:0/3230143798 conn(0x7f63a00f7100 msgr2=0x7f63a00f9510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:15.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.032+0000 7f63a7d14700 1 -- 172.21.15.46:0/3230143798 shutdown_connections 2023-06-23T01:02:15.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.032+0000 7f63a7d14700 1 -- 172.21.15.46:0/3230143798 wait complete. 2023-06-23T01:02:15.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.032+0000 7f63a7d14700 1 Processor -- start 2023-06-23T01:02:15.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.032+0000 7f63a7d14700 1 -- start start 2023-06-23T01:02:15.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.032+0000 7f63a7d14700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a00fb3b0 0x7f63a01149c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:15.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.032+0000 7f63a7d14700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a0114f00 con 0x7f63a00fb3b0 2023-06-23T01:02:15.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.032+0000 7f63a5ab0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a00fb3b0 0x7f63a01149c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:15.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.032+0000 7f63a5ab0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a00fb3b0 0x7f63a01149c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53648/0 (socket says 172.21.15.46:53648) 2023-06-23T01:02:15.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.032+0000 7f63a5ab0700 1 -- 172.21.15.46:0/336756487 learned_addr learned my addr 172.21.15.46:0/336756487 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:15.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.033+0000 7f63a5ab0700 1 -- 172.21.15.46:0/336756487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6398009940 con 0x7f63a00fb3b0 2023-06-23T01:02:15.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.033+0000 7f63a5ab0700 1 --2- 172.21.15.46:0/336756487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a00fb3b0 0x7f63a01149c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6398009110 tx=0x7f6398004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:15.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.034+0000 7f639e7fc700 1 -- 172.21.15.46:0/336756487 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f639800e040 con 0x7f63a00fb3b0 2023-06-23T01:02:15.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.034+0000 7f639e7fc700 1 -- 172.21.15.46:0/336756487 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6398008d60 con 0x7f63a00fb3b0 2023-06-23T01:02:15.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.034+0000 7f63a7d14700 1 -- 172.21.15.46:0/336756487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a0115100 con 0x7f63a00fb3b0 2023-06-23T01:02:15.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.034+0000 7f639e7fc700 1 -- 172.21.15.46:0/336756487 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f639801c400 con 0x7f63a00fb3b0 2023-06-23T01:02:15.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.035+0000 7f63a7d14700 1 -- 172.21.15.46:0/336756487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a010e880 con 0x7f63a00fb3b0 2023-06-23T01:02:15.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.035+0000 7f639e7fc700 1 -- 172.21.15.46:0/336756487 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f639801c5f0 con 0x7f63a00fb3b0 2023-06-23T01:02:15.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.035+0000 7f63a7d14700 1 -- 172.21.15.46:0/336756487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a005f060 con 0x7f63a00fb3b0 2023-06-23T01:02:15.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.036+0000 7f639e7fc700 1 --2- 172.21.15.46:0/336756487 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f63880366b0 0x7f6388038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:15.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.036+0000 7f639e7fc700 1 -- 172.21.15.46:0/336756487 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6398049f60 con 0x7f63a00fb3b0 2023-06-23T01:02:15.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.036+0000 7f63a52af700 1 --2- 172.21.15.46:0/336756487 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f63880366b0 0x7f6388038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:15.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.036+0000 7f63a52af700 1 --2- 172.21.15.46:0/336756487 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f63880366b0 0x7f6388038b60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6394009990 tx=0x7f6394006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:15.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.040+0000 7f639e7fc700 1 -- 172.21.15.46:0/336756487 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f639801a300 con 0x7f63a00fb3b0 2023-06-23T01:02:15.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.239+0000 7f63a7d14700 1 -- 172.21.15.46:0/336756487 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63a0114690 con 0x7f63880366b0 2023-06-23T01:02:15.244 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.241+0000 7f639e7fc700 1 -- 172.21.15.46:0/336756487 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f63a0114690 con 0x7f63880366b0 2023-06-23T01:02:15.245 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:15.245 INFO:teuthology.orchestra.run.smithi046.stdout:[{"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "172.21.15.92", "hostname": "smithi092", "labels": [], "status": ""}] 2023-06-23T01:02:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.243+0000 7f63a7d14700 1 -- 172.21.15.46:0/336756487 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f63880366b0 msgr2=0x7f6388038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.243+0000 7f63a7d14700 1 --2- 172.21.15.46:0/336756487 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f63880366b0 0x7f6388038b60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6394009990 tx=0x7f6394006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.243+0000 7f63a7d14700 1 -- 172.21.15.46:0/336756487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a00fb3b0 msgr2=0x7f63a01149c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.243+0000 7f63a7d14700 1 --2- 172.21.15.46:0/336756487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a00fb3b0 0x7f63a01149c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6398009110 tx=0x7f6398004d10 comp rx=0 tx=0).stop 2023-06-23T01:02:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.243+0000 7f63a7d14700 1 -- 172.21.15.46:0/336756487 shutdown_connections 2023-06-23T01:02:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.243+0000 7f63a7d14700 1 --2- 172.21.15.46:0/336756487 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f63880366b0 0x7f6388038b60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.243+0000 7f63a7d14700 1 --2- 172.21.15.46:0/336756487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a00fb3b0 0x7f63a01149c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.243+0000 7f63a7d14700 1 -- 172.21.15.46:0/336756487 >> 172.21.15.46:0/336756487 conn(0x7f63a00f7100 msgr2=0x7f63a00f8940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.243+0000 7f63a7d14700 1 -- 172.21.15.46:0/336756487 shutdown_connections 2023-06-23T01:02:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:15.243+0000 7f63a7d14700 1 -- 172.21.15.46:0/336756487 wait complete. 2023-06-23T01:02:15.270 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:15 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:15.578 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi131 2023-06-23T01:02:15.579 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-06-23T01:02:15.579 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/etc/ceph/ceph.conf 2023-06-23T01:02:15.596 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-06-23T01:02:15.597 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-06-23T01:02:15.653 INFO:tasks.cephadm:Adding host smithi131 to orchestrator... 2023-06-23T01:02:15.654 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch host add smithi131 2023-06-23T01:02:16.248 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:02:16.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:16 smithi046 ceph-mon[84949]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-06-23T01:02:16.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:16.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:16 smithi046 ceph-mon[84949]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:17.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.319+0000 7f3c66f7a700 1 -- 172.21.15.46:0/1485266078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c60101600 msgr2=0x7f3c601019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:17.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.319+0000 7f3c66f7a700 1 --2- 172.21.15.46:0/1485266078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c60101600 0x7f3c601019d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3c540048c0 tx=0x7f3c54009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:17.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.319+0000 7f3c66f7a700 1 -- 172.21.15.46:0/1485266078 shutdown_connections 2023-06-23T01:02:17.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.319+0000 7f3c66f7a700 1 --2- 172.21.15.46:0/1485266078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c60101600 0x7f3c601019d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:17.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.319+0000 7f3c66f7a700 1 -- 172.21.15.46:0/1485266078 >> 172.21.15.46:0/1485266078 conn(0x7f3c60071b80 msgr2=0x7f3c60071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:17.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.319+0000 7f3c66f7a700 1 -- 172.21.15.46:0/1485266078 shutdown_connections 2023-06-23T01:02:17.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.319+0000 7f3c66f7a700 1 -- 172.21.15.46:0/1485266078 wait complete. 2023-06-23T01:02:17.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.319+0000 7f3c66f7a700 1 Processor -- start 2023-06-23T01:02:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.319+0000 7f3c66f7a700 1 -- start start 2023-06-23T01:02:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.320+0000 7f3c66f7a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c60101600 0x7f3c6010d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.320+0000 7f3c66f7a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6010dd90 con 0x7f3c60101600 2023-06-23T01:02:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.320+0000 7f3c64d16700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c60101600 0x7f3c6010d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.320+0000 7f3c64d16700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c60101600 0x7f3c6010d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53664/0 (socket says 172.21.15.46:53664) 2023-06-23T01:02:17.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.320+0000 7f3c64d16700 1 -- 172.21.15.46:0/168499139 learned_addr learned my addr 172.21.15.46:0/168499139 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:17.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.320+0000 7f3c64d16700 1 -- 172.21.15.46:0/168499139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c54005040 con 0x7f3c60101600 2023-06-23T01:02:17.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.320+0000 7f3c64d16700 1 --2- 172.21.15.46:0/168499139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c60101600 0x7f3c6010d850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c54008f00 tx=0x7f3c54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:17.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.320+0000 7f3c5d7fa700 1 -- 172.21.15.46:0/168499139 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c5400e040 con 0x7f3c60101600 2023-06-23T01:02:17.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.320+0000 7f3c5d7fa700 1 -- 172.21.15.46:0/168499139 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3c5400be50 con 0x7f3c60101600 2023-06-23T01:02:17.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.320+0000 7f3c66f7a700 1 -- 172.21.15.46:0/168499139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6010df90 con 0x7f3c60101600 2023-06-23T01:02:17.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.320+0000 7f3c5d7fa700 1 -- 172.21.15.46:0/168499139 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c54013d10 con 0x7f3c60101600 2023-06-23T01:02:17.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.321+0000 7f3c66f7a700 1 -- 172.21.15.46:0/168499139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6010e480 con 0x7f3c60101600 2023-06-23T01:02:17.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.321+0000 7f3c5d7fa700 1 -- 172.21.15.46:0/168499139 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f3c54025400 con 0x7f3c60101600 2023-06-23T01:02:17.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.321+0000 7f3c5d7fa700 1 --2- 172.21.15.46:0/168499139 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3c500366b0 0x7f3c50038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:17.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.321+0000 7f3c5d7fa700 1 -- 172.21.15.46:0/168499139 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3c54049500 con 0x7f3c60101600 2023-06-23T01:02:17.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.322+0000 7f3c66f7a700 1 -- 172.21.15.46:0/168499139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c60100900 con 0x7f3c60101600 2023-06-23T01:02:17.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.322+0000 7f3c5ffff700 1 --2- 172.21.15.46:0/168499139 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3c500366b0 0x7f3c50038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:17.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.322+0000 7f3c5ffff700 1 --2- 172.21.15.46:0/168499139 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3c500366b0 0x7f3c50038b60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009990 tx=0x7f3c4c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:17.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.326+0000 7f3c5d7fa700 1 -- 172.21.15.46:0/168499139 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f3c54011960 con 0x7f3c60101600 2023-06-23T01:02:17.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:17.527+0000 7f3c66f7a700 1 -- 172.21.15.46:0/168499139 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi131", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c60105750 con 0x7f3c500366b0 2023-06-23T01:02:19.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:18 smithi046 ceph-mon[84949]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi131", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:19.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:18 smithi046 ceph-mon[84949]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:19.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:19.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:19.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:19.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:19.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:02:19.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:19.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:19.364 INFO:teuthology.orchestra.run.smithi046.stdout:Added host 'smithi131' with addr '172.21.15.131' 2023-06-23T01:02:19.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:19.361+0000 7f3c5d7fa700 1 -- 172.21.15.46:0/168499139 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f3c60105750 con 0x7f3c500366b0 2023-06-23T01:02:19.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:19.363+0000 7f3c66f7a700 1 -- 172.21.15.46:0/168499139 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3c500366b0 msgr2=0x7f3c50038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:19.363+0000 7f3c66f7a700 1 --2- 172.21.15.46:0/168499139 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3c500366b0 0x7f3c50038b60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009990 tx=0x7f3c4c006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:19.363+0000 7f3c66f7a700 1 -- 172.21.15.46:0/168499139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c60101600 msgr2=0x7f3c6010d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:19.363+0000 7f3c66f7a700 1 --2- 172.21.15.46:0/168499139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c60101600 0x7f3c6010d850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c54008f00 tx=0x7f3c54009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:19.363+0000 7f3c66f7a700 1 -- 172.21.15.46:0/168499139 shutdown_connections 2023-06-23T01:02:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:19.363+0000 7f3c66f7a700 1 --2- 172.21.15.46:0/168499139 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f3c500366b0 0x7f3c50038b60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:19.363+0000 7f3c66f7a700 1 --2- 172.21.15.46:0/168499139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c60101600 0x7f3c6010d850 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:19.363+0000 7f3c66f7a700 1 -- 172.21.15.46:0/168499139 >> 172.21.15.46:0/168499139 conn(0x7f3c60071b80 msgr2=0x7f3c60102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:19.363+0000 7f3c66f7a700 1 -- 172.21.15.46:0/168499139 shutdown_connections 2023-06-23T01:02:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:19.363+0000 7f3c66f7a700 1 -- 172.21.15.46:0/168499139 wait complete. 2023-06-23T01:02:19.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch host ls --format=json 2023-06-23T01:02:20.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:19 smithi046 ceph-mon[84949]: Updating smithi092:/etc/ceph/ceph.conf 2023-06-23T01:02:20.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:19 smithi046 ceph-mon[84949]: Deploying cephadm binary to smithi131 2023-06-23T01:02:20.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:19 smithi046 ceph-mon[84949]: Updating smithi092:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:20.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:19 smithi046 ceph-mon[84949]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2023-06-23T01:02:20.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:19 smithi046 ceph-mon[84949]: Updating smithi092:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.client.admin.keyring 2023-06-23T01:02:20.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:20.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:20.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:20.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:20.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:20.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:20.393 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:02:21.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:20 smithi046 ceph-mon[84949]: Added host smithi131 2023-06-23T01:02:21.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:20 smithi046 ceph-mon[84949]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:21.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:20 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:21.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.813+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/2562009732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6c41026d0 msgr2=0x7fe6c4102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:21.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.813+0000 7fe6cc2ac700 1 --2- 172.21.15.46:0/2562009732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6c41026d0 0x7fe6c4102aa0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0048c0 tx=0x7fe6bc009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:21.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.813+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/2562009732 shutdown_connections 2023-06-23T01:02:21.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.813+0000 7fe6cc2ac700 1 --2- 172.21.15.46:0/2562009732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6c41026d0 0x7fe6c4102aa0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.813+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/2562009732 >> 172.21.15.46:0/2562009732 conn(0x7fe6c4071b50 msgr2=0x7fe6c4071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.813+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/2562009732 shutdown_connections 2023-06-23T01:02:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.813+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/2562009732 wait complete. 2023-06-23T01:02:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.813+0000 7fe6cc2ac700 1 Processor -- start 2023-06-23T01:02:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.813+0000 7fe6cc2ac700 1 -- start start 2023-06-23T01:02:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.813+0000 7fe6cc2ac700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6c41026d0 0x7fe6c41127f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.813+0000 7fe6cc2ac700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c4112d30 con 0x7fe6c41026d0 2023-06-23T01:02:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.814+0000 7fe6ca048700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6c41026d0 0x7fe6c41127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.814+0000 7fe6ca048700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6c41026d0 0x7fe6c41127f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53694/0 (socket says 172.21.15.46:53694) 2023-06-23T01:02:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.814+0000 7fe6ca048700 1 -- 172.21.15.46:0/320454982 learned_addr learned my addr 172.21.15.46:0/320454982 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.814+0000 7fe6ca048700 1 -- 172.21.15.46:0/320454982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc005040 con 0x7fe6c41026d0 2023-06-23T01:02:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.814+0000 7fe6ca048700 1 --2- 172.21.15.46:0/320454982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6c41026d0 0x7fe6c41127f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009b80 tx=0x7fe6bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.814+0000 7fe6c2ffd700 1 -- 172.21.15.46:0/320454982 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6bc00e040 con 0x7fe6c41026d0 2023-06-23T01:02:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.814+0000 7fe6c2ffd700 1 -- 172.21.15.46:0/320454982 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe6bc00bd30 con 0x7fe6c41026d0 2023-06-23T01:02:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.814+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/320454982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c4112f30 con 0x7fe6c41026d0 2023-06-23T01:02:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.814+0000 7fe6c2ffd700 1 -- 172.21.15.46:0/320454982 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6bc013d80 con 0x7fe6c41026d0 2023-06-23T01:02:21.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.814+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/320454982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c41133a0 con 0x7fe6c41026d0 2023-06-23T01:02:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.815+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/320454982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c405f060 con 0x7fe6c41026d0 2023-06-23T01:02:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.815+0000 7fe6c2ffd700 1 -- 172.21.15.46:0/320454982 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fe6bc00b890 con 0x7fe6c41026d0 2023-06-23T01:02:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.816+0000 7fe6c2ffd700 1 --2- 172.21.15.46:0/320454982 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe6ac036660 0x7fe6ac038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.816+0000 7fe6c2ffd700 1 -- 172.21.15.46:0/320454982 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe6bc0494c0 con 0x7fe6c41026d0 2023-06-23T01:02:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.816+0000 7fe6c9847700 1 --2- 172.21.15.46:0/320454982 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe6ac036660 0x7fe6ac038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:21.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.816+0000 7fe6c9847700 1 --2- 172.21.15.46:0/320454982 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe6ac036660 0x7fe6ac038b10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009990 tx=0x7fe6b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:21.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:21.820+0000 7fe6c2ffd700 1 -- 172.21.15.46:0/320454982 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fe6bc011760 con 0x7fe6c41026d0 2023-06-23T01:02:22.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.024+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/320454982 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe6c404b9e0 con 0x7fe6ac036660 2023-06-23T01:02:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.025+0000 7fe6c2ffd700 1 -- 172.21.15.46:0/320454982 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7fe6c404b9e0 con 0x7fe6ac036660 2023-06-23T01:02:22.029 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:02:22.029 INFO:teuthology.orchestra.run.smithi046.stdout:[{"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "172.21.15.92", "hostname": "smithi092", "labels": [], "status": ""}, {"addr": "172.21.15.131", "hostname": "smithi131", "labels": [], "status": ""}] 2023-06-23T01:02:22.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.027+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/320454982 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe6ac036660 msgr2=0x7fe6ac038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:22.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.027+0000 7fe6cc2ac700 1 --2- 172.21.15.46:0/320454982 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe6ac036660 0x7fe6ac038b10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009990 tx=0x7fe6b8006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:22.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.027+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/320454982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6c41026d0 msgr2=0x7fe6c41127f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:22.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.027+0000 7fe6cc2ac700 1 --2- 172.21.15.46:0/320454982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6c41026d0 0x7fe6c41127f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009b80 tx=0x7fe6bc009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:22.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.028+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/320454982 shutdown_connections 2023-06-23T01:02:22.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.028+0000 7fe6cc2ac700 1 --2- 172.21.15.46:0/320454982 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe6ac036660 0x7fe6ac038b10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:22.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.028+0000 7fe6cc2ac700 1 --2- 172.21.15.46:0/320454982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6c41026d0 0x7fe6c41127f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:22.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.028+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/320454982 >> 172.21.15.46:0/320454982 conn(0x7fe6c4071b50 msgr2=0x7fe6c4103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:22.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.028+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/320454982 shutdown_connections 2023-06-23T01:02:22.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:22.028+0000 7fe6cc2ac700 1 -- 172.21.15.46:0/320454982 wait complete. 2023-06-23T01:02:22.442 INFO:tasks.cephadm:Setting crush tunables to default 2023-06-23T01:02:22.443 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph osd crush tunables default 2023-06-23T01:02:23.036 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:02:23.209 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:22 smithi046 ceph-mon[84949]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:23.209 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:22 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.120+0000 7f5fab5a3700 1 -- 172.21.15.46:0/590267490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa41015b0 msgr2=0x7f5fa4101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.120+0000 7f5fab5a3700 1 --2- 172.21.15.46:0/590267490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa41015b0 0x7f5fa4101980 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00048c0 tx=0x7f5fa0009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.120+0000 7f5fab5a3700 1 -- 172.21.15.46:0/590267490 shutdown_connections 2023-06-23T01:02:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.120+0000 7f5fab5a3700 1 --2- 172.21.15.46:0/590267490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa41015b0 0x7f5fa4101980 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.120+0000 7f5fab5a3700 1 -- 172.21.15.46:0/590267490 >> 172.21.15.46:0/590267490 conn(0x7f5fa40713d0 msgr2=0x7f5fa40717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.120+0000 7f5fab5a3700 1 -- 172.21.15.46:0/590267490 shutdown_connections 2023-06-23T01:02:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.120+0000 7f5fab5a3700 1 -- 172.21.15.46:0/590267490 wait complete. 2023-06-23T01:02:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.121+0000 7f5fab5a3700 1 Processor -- start 2023-06-23T01:02:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.121+0000 7f5fab5a3700 1 -- start start 2023-06-23T01:02:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.121+0000 7f5fab5a3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa41015b0 0x7f5fa410d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.121+0000 7f5fab5a3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa410dd10 con 0x7f5fa41015b0 2023-06-23T01:02:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.121+0000 7f5fa933f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa41015b0 0x7f5fa410d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.121+0000 7f5fa933f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa41015b0 0x7f5fa410d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53706/0 (socket says 172.21.15.46:53706) 2023-06-23T01:02:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.121+0000 7f5fa933f700 1 -- 172.21.15.46:0/4114537337 learned_addr learned my addr 172.21.15.46:0/4114537337 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.121+0000 7f5fa933f700 1 -- 172.21.15.46:0/4114537337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa0005040 con 0x7f5fa41015b0 2023-06-23T01:02:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.121+0000 7f5fa933f700 1 --2- 172.21.15.46:0/4114537337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa41015b0 0x7f5fa410d7d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009b80 tx=0x7f5fa0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.122+0000 7f5f99ffb700 1 -- 172.21.15.46:0/4114537337 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5fa000e040 con 0x7f5fa41015b0 2023-06-23T01:02:24.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.122+0000 7f5f99ffb700 1 -- 172.21.15.46:0/4114537337 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5fa000bd30 con 0x7f5fa41015b0 2023-06-23T01:02:24.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.122+0000 7f5fab5a3700 1 -- 172.21.15.46:0/4114537337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa410df10 con 0x7f5fa41015b0 2023-06-23T01:02:24.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.122+0000 7f5f99ffb700 1 -- 172.21.15.46:0/4114537337 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5fa0013d80 con 0x7f5fa41015b0 2023-06-23T01:02:24.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.122+0000 7f5fab5a3700 1 -- 172.21.15.46:0/4114537337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa410e400 con 0x7f5fa41015b0 2023-06-23T01:02:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.123+0000 7f5f99ffb700 1 -- 172.21.15.46:0/4114537337 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f5fa000b890 con 0x7f5fa41015b0 2023-06-23T01:02:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.123+0000 7f5f99ffb700 1 --2- 172.21.15.46:0/4114537337 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5f90036660 0x7f5f90038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.123+0000 7f5fab5a3700 1 -- 172.21.15.46:0/4114537337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa4100820 con 0x7f5fa41015b0 2023-06-23T01:02:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.123+0000 7f5f99ffb700 1 -- 172.21.15.46:0/4114537337 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5fa00494c0 con 0x7f5fa41015b0 2023-06-23T01:02:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.123+0000 7f5fa8b3e700 1 --2- 172.21.15.46:0/4114537337 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5f90036660 0x7f5f90038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.123+0000 7f5fa8b3e700 1 --2- 172.21.15.46:0/4114537337 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5f90036660 0x7f5f90038b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5f94009990 tx=0x7f5f94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:24.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.128+0000 7f5f99ffb700 1 -- 172.21.15.46:0/4114537337 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f5fa0025400 con 0x7f5fa41015b0 2023-06-23T01:02:24.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:23 smithi046 ceph-mon[84949]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-06-23T01:02:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.331+0000 7f5fab5a3700 1 -- 172.21.15.46:0/4114537337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5fa404b970 con 0x7f5fa41015b0 2023-06-23T01:02:24.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.832+0000 7f5f99ffb700 1 -- 172.21.15.46:0/4114537337 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5fa003d270 con 0x7f5fa41015b0 2023-06-23T01:02:24.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.835+0000 7f5fab5a3700 1 -- 172.21.15.46:0/4114537337 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5f90036660 msgr2=0x7f5f90038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.835+0000 7f5fab5a3700 1 --2- 172.21.15.46:0/4114537337 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5f90036660 0x7f5f90038b10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5f94009990 tx=0x7f5f94006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.835+0000 7f5fab5a3700 1 -- 172.21.15.46:0/4114537337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa41015b0 msgr2=0x7f5fa410d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.835+0000 7f5fab5a3700 1 --2- 172.21.15.46:0/4114537337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa41015b0 0x7f5fa410d7d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009b80 tx=0x7f5fa0009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.835+0000 7f5fab5a3700 1 -- 172.21.15.46:0/4114537337 shutdown_connections 2023-06-23T01:02:24.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.835+0000 7f5fab5a3700 1 --2- 172.21.15.46:0/4114537337 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5f90036660 0x7f5f90038b10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.835+0000 7f5fab5a3700 1 --2- 172.21.15.46:0/4114537337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fa41015b0 0x7f5fa410d7d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.835+0000 7f5fab5a3700 1 -- 172.21.15.46:0/4114537337 >> 172.21.15.46:0/4114537337 conn(0x7f5fa40713d0 msgr2=0x7f5fa40f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.835+0000 7f5fab5a3700 1 -- 172.21.15.46:0/4114537337 shutdown_connections 2023-06-23T01:02:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:24.835+0000 7f5fab5a3700 1 -- 172.21.15.46:0/4114537337 wait complete. 2023-06-23T01:02:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2023-06-23T01:02:25.166 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:24 smithi046 ceph-mon[84949]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:25.166 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:24 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/4114537337' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-06-23T01:02:25.166 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:25.166 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:25.167 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:25.167 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:25.167 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:02:25.167 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:25.167 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:25.185 INFO:tasks.cephadm:Adding mon.a on smithi046 2023-06-23T01:02:25.185 INFO:tasks.cephadm:Adding mon.b on smithi092 2023-06-23T01:02:25.186 INFO:tasks.cephadm:Adding mon.c on smithi131 2023-06-23T01:02:25.186 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch apply mon '3;smithi046:172.21.15.46=a;smithi092:172.21.15.92=b;smithi131:172.21.15.131=c' 2023-06-23T01:02:25.799 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:26.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:25 smithi046 ceph-mon[84949]: Updating smithi131:/etc/ceph/ceph.conf 2023-06-23T01:02:26.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:25 smithi046 ceph-mon[84949]: Updating smithi131:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:26.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:25 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/4114537337' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-06-23T01:02:26.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:25 smithi046 ceph-mon[84949]: osdmap e4: 0 total, 0 up, 0 in 2023-06-23T01:02:26.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:25 smithi046 ceph-mon[84949]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2023-06-23T01:02:26.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:25 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:26.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:25 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:26.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:25 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:26.898 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.895+0000 7f055c65d700 1 -- 172.21.15.131:0/2318885659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05541026c0 msgr2=0x7f0554102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:26.899 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.895+0000 7f055c65d700 1 --2- 172.21.15.131:0/2318885659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05541026c0 0x7f0554102a90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0544009c60 tx=0x7f0544004d10 comp rx=0 tx=0).stop 2023-06-23T01:02:26.899 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.895+0000 7f055c65d700 1 -- 172.21.15.131:0/2318885659 shutdown_connections 2023-06-23T01:02:26.899 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.895+0000 7f055c65d700 1 --2- 172.21.15.131:0/2318885659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05541026c0 0x7f0554102a90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:26.899 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.895+0000 7f055c65d700 1 -- 172.21.15.131:0/2318885659 >> 172.21.15.131:0/2318885659 conn(0x7f05540f8200 msgr2=0x7f05540fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:26.899 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.895+0000 7f055c65d700 1 -- 172.21.15.131:0/2318885659 shutdown_connections 2023-06-23T01:02:26.900 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.895+0000 7f055c65d700 1 -- 172.21.15.131:0/2318885659 wait complete. 2023-06-23T01:02:26.900 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.895+0000 7f055c65d700 1 Processor -- start 2023-06-23T01:02:26.900 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.895+0000 7f055c65d700 1 -- start start 2023-06-23T01:02:26.900 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.895+0000 7f055c65d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05541026c0 0x7f05541127e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:26.900 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.895+0000 7f055c65d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0554112d20 con 0x7f05541026c0 2023-06-23T01:02:26.900 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.896+0000 7f055a3f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05541026c0 0x7f05541127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:26.900 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.896+0000 7f055a3f9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05541026c0 0x7f05541127e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.131:60746/0 (socket says 172.21.15.131:60746) 2023-06-23T01:02:26.901 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.896+0000 7f055a3f9700 1 -- 172.21.15.131:0/2508383565 learned_addr learned my addr 172.21.15.131:0/2508383565 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-06-23T01:02:26.902 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.896+0000 7f055a3f9700 1 -- 172.21.15.131:0/2508383565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0544009940 con 0x7f05541026c0 2023-06-23T01:02:26.902 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.896+0000 7f055a3f9700 1 --2- 172.21.15.131:0/2508383565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05541026c0 0x7f05541127e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0544009110 tx=0x7f0544004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:26.902 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.896+0000 7f0552ffd700 1 -- 172.21.15.131:0/2508383565 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f054400e040 con 0x7f05541026c0 2023-06-23T01:02:26.902 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.896+0000 7f055c65d700 1 -- 172.21.15.131:0/2508383565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0554112f20 con 0x7f05541026c0 2023-06-23T01:02:26.902 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.896+0000 7f0552ffd700 1 -- 172.21.15.131:0/2508383565 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0544008d60 con 0x7f05541026c0 2023-06-23T01:02:26.903 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.896+0000 7f0552ffd700 1 -- 172.21.15.131:0/2508383565 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f054401c400 con 0x7f05541026c0 2023-06-23T01:02:26.903 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.896+0000 7f055c65d700 1 -- 172.21.15.131:0/2508383565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055410c6a0 con 0x7f05541026c0 2023-06-23T01:02:26.903 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.897+0000 7f0552ffd700 1 -- 172.21.15.131:0/2508383565 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f054401c640 con 0x7f05541026c0 2023-06-23T01:02:26.903 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.897+0000 7f0552ffd700 1 --2- 172.21.15.131:0/2508383565 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f053c036660 0x7f053c038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:26.903 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.897+0000 7f055c65d700 1 -- 172.21.15.131:0/2508383565 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f055405f060 con 0x7f05541026c0 2023-06-23T01:02:26.903 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.897+0000 7f0552ffd700 1 -- 172.21.15.131:0/2508383565 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0544049e90 con 0x7f05541026c0 2023-06-23T01:02:26.903 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.898+0000 7f0559bf8700 1 --2- 172.21.15.131:0/2508383565 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f053c036660 0x7f053c038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:26.904 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.898+0000 7f0559bf8700 1 --2- 172.21.15.131:0/2508383565 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f053c036660 0x7f053c038b10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0548009990 tx=0x7f0548006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:26.906 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:26.903+0000 7f0552ffd700 1 -- 172.21.15.131:0/2508383565 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f0544017080 con 0x7f05541026c0 2023-06-23T01:02:27.099 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.095+0000 7f055c65d700 1 -- 172.21.15.131:0/2508383565 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi092:172.21.15.92=b;smithi131:172.21.15.131=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f055410ddc0 con 0x7f053c036660 2023-06-23T01:02:27.105 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.103+0000 7f0552ffd700 1 -- 172.21.15.131:0/2508383565 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f055410ddc0 con 0x7f053c036660 2023-06-23T01:02:27.105 INFO:teuthology.orchestra.run.smithi131.stdout:Scheduled mon update... 2023-06-23T01:02:27.107 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.104+0000 7f055c65d700 1 -- 172.21.15.131:0/2508383565 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f053c036660 msgr2=0x7f053c038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:27.107 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.104+0000 7f055c65d700 1 --2- 172.21.15.131:0/2508383565 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f053c036660 0x7f053c038b10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0548009990 tx=0x7f0548006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:27.107 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.104+0000 7f055c65d700 1 -- 172.21.15.131:0/2508383565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05541026c0 msgr2=0x7f05541127e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:27.107 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.104+0000 7f055c65d700 1 --2- 172.21.15.131:0/2508383565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05541026c0 0x7f05541127e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0544009110 tx=0x7f0544004d10 comp rx=0 tx=0).stop 2023-06-23T01:02:27.107 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.104+0000 7f055c65d700 1 -- 172.21.15.131:0/2508383565 shutdown_connections 2023-06-23T01:02:27.108 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.104+0000 7f055c65d700 1 --2- 172.21.15.131:0/2508383565 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f053c036660 0x7f053c038b10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:27.108 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.104+0000 7f055c65d700 1 --2- 172.21.15.131:0/2508383565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05541026c0 0x7f05541127e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:27.108 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.104+0000 7f055c65d700 1 -- 172.21.15.131:0/2508383565 >> 172.21.15.131:0/2508383565 conn(0x7f05540f8200 msgr2=0x7f05540f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:27.108 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.104+0000 7f055c65d700 1 -- 172.21.15.131:0/2508383565 shutdown_connections 2023-06-23T01:02:27.108 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:27.104+0000 7f055c65d700 1 -- 172.21.15.131:0/2508383565 wait complete. 2023-06-23T01:02:27.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:26 smithi046 ceph-mon[84949]: Updating smithi131:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.client.admin.keyring 2023-06-23T01:02:27.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:26 smithi046 ceph-mon[84949]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:27.474 DEBUG:teuthology.orchestra.run.smithi092:mon.b> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.b.service 2023-06-23T01:02:27.476 DEBUG:teuthology.orchestra.run.smithi131:mon.c> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.c.service 2023-06-23T01:02:27.478 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-06-23T01:02:27.479 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph mon dump -f json 2023-06-23T01:02:27.505 INFO:journalctl@ceph.mon.c.smithi131.stdout:-- Logs begin at Fri 2023-06-23 00:44:02 UTC. -- 2023-06-23T01:02:27.509 INFO:journalctl@ceph.mon.b.smithi092.stdout:-- Logs begin at Fri 2023-06-23 00:44:08 UTC. -- 2023-06-23T01:02:28.429 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:28.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:28 smithi046 ceph-mon[84949]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi092:172.21.15.92=b;smithi131:172.21.15.131=c", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:28.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:28 smithi046 ceph-mon[84949]: Saving service mon spec with placement smithi046:172.21.15.46=a;smithi092:172.21.15.92=b;smithi131:172.21.15.131=c;count:3 2023-06-23T01:02:28.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:28.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:28 smithi046 ceph-mon[84949]: Deploying daemon mon.c on smithi131 2023-06-23T01:02:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:28 smithi046 ceph-mon[84949]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:30.143 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.138+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/3652177719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ee4100cf0 msgr2=0x7f5ee41010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:30.143 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.138+0000 7f5eeb5b1700 1 --2- 172.21.15.131:0/3652177719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ee4100cf0 0x7f5ee41010c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5ed40048c0 tx=0x7f5ed4009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:30.143 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.138+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/3652177719 shutdown_connections 2023-06-23T01:02:30.143 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.138+0000 7f5eeb5b1700 1 --2- 172.21.15.131:0/3652177719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ee4100cf0 0x7f5ee41010c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:30.144 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.138+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/3652177719 >> 172.21.15.131:0/3652177719 conn(0x7f5ee4071b50 msgr2=0x7f5ee4071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:30.144 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.138+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/3652177719 shutdown_connections 2023-06-23T01:02:30.144 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.138+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/3652177719 wait complete. 2023-06-23T01:02:30.145 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.139+0000 7f5eeb5b1700 1 Processor -- start 2023-06-23T01:02:30.145 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.139+0000 7f5eeb5b1700 1 -- start start 2023-06-23T01:02:30.145 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.139+0000 7f5eeb5b1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ee4100cf0 0x7f5ee41116b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:30.145 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.139+0000 7f5eeb5b1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee4111bf0 con 0x7f5ee4100cf0 2023-06-23T01:02:30.145 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.139+0000 7f5ee934d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ee4100cf0 0x7f5ee41116b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:30.145 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.139+0000 7f5ee934d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ee4100cf0 0x7f5ee41116b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.131:60750/0 (socket says 172.21.15.131:60750) 2023-06-23T01:02:30.146 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.139+0000 7f5ee934d700 1 -- 172.21.15.131:0/1646814983 learned_addr learned my addr 172.21.15.131:0/1646814983 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-06-23T01:02:30.146 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.139+0000 7f5ee934d700 1 -- 172.21.15.131:0/1646814983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed4005040 con 0x7f5ee4100cf0 2023-06-23T01:02:30.146 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.140+0000 7f5ee934d700 1 --2- 172.21.15.131:0/1646814983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ee4100cf0 0x7f5ee41116b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009aa0 tx=0x7f5ed4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:30.146 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.140+0000 7f5ed9ffb700 1 -- 172.21.15.131:0/1646814983 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ed400e040 con 0x7f5ee4100cf0 2023-06-23T01:02:30.146 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.140+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/1646814983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee4111df0 con 0x7f5ee4100cf0 2023-06-23T01:02:30.147 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.140+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/1646814983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee4112280 con 0x7f5ee4100cf0 2023-06-23T01:02:30.147 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.140+0000 7f5ed9ffb700 1 -- 172.21.15.131:0/1646814983 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5ed4028460 con 0x7f5ee4100cf0 2023-06-23T01:02:30.147 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.140+0000 7f5ed9ffb700 1 -- 172.21.15.131:0/1646814983 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ed401e420 con 0x7f5ee4100cf0 2023-06-23T01:02:30.148 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.141+0000 7f5ed9ffb700 1 -- 172.21.15.131:0/1646814983 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f5ed401e690 con 0x7f5ee4100cf0 2023-06-23T01:02:30.148 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.141+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/1646814983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee4100010 con 0x7f5ee4100cf0 2023-06-23T01:02:30.148 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.141+0000 7f5ed9ffb700 1 --2- 172.21.15.131:0/1646814983 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5ed0036660 0x7f5ed0038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:30.148 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.141+0000 7f5ed9ffb700 1 -- 172.21.15.131:0/1646814983 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5ed404c0c0 con 0x7f5ee4100cf0 2023-06-23T01:02:30.148 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.142+0000 7f5ee8b4c700 1 --2- 172.21.15.131:0/1646814983 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5ed0036660 0x7f5ed0038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:30.148 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.142+0000 7f5ee8b4c700 1 --2- 172.21.15.131:0/1646814983 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5ed0036660 0x7f5ed0038b10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009990 tx=0x7f5ee0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:30.149 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.147+0000 7f5ed9ffb700 1 -- 172.21.15.131:0/1646814983 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f5ed4026360 con 0x7f5ee4100cf0 2023-06-23T01:02:30.440 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.437+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/1646814983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ee4104e60 con 0x7f5ee4100cf0 2023-06-23T01:02:30.441 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.439+0000 7f5ed9ffb700 1 -- 172.21.15.131:0/1646814983 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+731 (secure 0 0 0) 0x7f5ed40285d0 con 0x7f5ee4100cf0 2023-06-23T01:02:30.441 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T01:02:30.441 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":1,"fsid":"458642ee-1161-11ee-9b2b-001a4aab830c","modified":"2023-06-23T01:00:55.582541Z","created":"2023-06-23T01:00:55.582541Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-06-23T01:02:30.443 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.440+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/1646814983 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5ed0036660 msgr2=0x7f5ed0038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:30.443 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.440+0000 7f5eeb5b1700 1 --2- 172.21.15.131:0/1646814983 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5ed0036660 0x7f5ed0038b10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009990 tx=0x7f5ee0006e40 comp rx=0 tx=0).stop 2023-06-23T01:02:30.443 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.440+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/1646814983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ee4100cf0 msgr2=0x7f5ee41116b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:30.444 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.440+0000 7f5eeb5b1700 1 --2- 172.21.15.131:0/1646814983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ee4100cf0 0x7f5ee41116b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5ed4009aa0 tx=0x7f5ed4009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:30.444 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.440+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/1646814983 shutdown_connections 2023-06-23T01:02:30.444 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.440+0000 7f5eeb5b1700 1 --2- 172.21.15.131:0/1646814983 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5ed0036660 0x7f5ed0038b10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:30.444 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.440+0000 7f5eeb5b1700 1 --2- 172.21.15.131:0/1646814983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ee4100cf0 0x7f5ee41116b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:30.444 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.440+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/1646814983 >> 172.21.15.131:0/1646814983 conn(0x7f5ee4071b50 msgr2=0x7f5ee40f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:30.444 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.440+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/1646814983 shutdown_connections 2023-06-23T01:02:30.445 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:30.440+0000 7f5eeb5b1700 1 -- 172.21.15.131:0/1646814983 wait complete. 2023-06-23T01:02:30.445 INFO:teuthology.orchestra.run.smithi131.stderr:dumped monmap epoch 1 2023-06-23T01:02:30.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:30 smithi131 ceph-mon[86495]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-06-23T01:02:30.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:30 smithi131 systemd[1]: Started Ceph mon.c for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:02:32.016 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-06-23T01:02:32.016 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph mon dump -f json 2023-06-23T01:02:32.605 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:02:36.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:02:36.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: mon.a calling monitor election 2023-06-23T01:02:36.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:36.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: mon.c calling monitor election 2023-06-23T01:02:36.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:36.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-23T01:02:36.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: monmap e2: 2 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],c=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} removed_ranks: {} 2023-06-23T01:02:36.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: fsmap 2023-06-23T01:02:36.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: osdmap e4: 0 total, 0 up, 0 in 2023-06-23T01:02:36.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: mgrmap e13: x(active, since 39s) 2023-06-23T01:02:36.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: overall HEALTH_OK 2023-06-23T01:02:36.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:36.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:36.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:36.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:36.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:36.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:02:36.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: mon.a calling monitor election 2023-06-23T01:02:36.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:36.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: mon.c calling monitor election 2023-06-23T01:02:36.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:36.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:36.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-23T01:02:36.236 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: monmap e2: 2 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],c=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} removed_ranks: {} 2023-06-23T01:02:36.236 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: fsmap 2023-06-23T01:02:36.236 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: osdmap e4: 0 total, 0 up, 0 in 2023-06-23T01:02:36.236 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: mgrmap e13: x(active, since 39s) 2023-06-23T01:02:36.237 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: overall HEALTH_OK 2023-06-23T01:02:36.237 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:36.237 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:36.238 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:36.238 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:36.238 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:36.818 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.814+0000 7f03f990d700 1 -- 172.21.15.131:0/3974447679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03e0003580 msgr2=0x7f03e0005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:36.818 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.814+0000 7f03f990d700 1 --2- 172.21.15.131:0/3974447679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03e0003580 0x7f03e0005a00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f03f40fc6b0 tx=0x7f03e400b040 comp rx=0 tx=0).stop 2023-06-23T01:02:36.818 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.814+0000 7f03f990d700 1 -- 172.21.15.131:0/3974447679 shutdown_connections 2023-06-23T01:02:36.818 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.814+0000 7f03f990d700 1 --2- 172.21.15.131:0/3974447679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03e0003580 0x7f03e0005a00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:36.819 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.814+0000 7f03f990d700 1 -- 172.21.15.131:0/3974447679 >> 172.21.15.131:0/3974447679 conn(0x7f03f40f8300 msgr2=0x7f03f40fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:36.819 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.814+0000 7f03f990d700 1 -- 172.21.15.131:0/3974447679 shutdown_connections 2023-06-23T01:02:36.819 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.814+0000 7f03f990d700 1 -- 172.21.15.131:0/3974447679 wait complete. 2023-06-23T01:02:36.819 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03f990d700 1 Processor -- start 2023-06-23T01:02:36.819 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03f990d700 1 -- start start 2023-06-23T01:02:36.819 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03f990d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03e0003580 0x7f03f4114cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:36.819 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03f990d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f4115200 0x7f03f410eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:36.819 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03f990d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f4115780 con 0x7f03e0003580 2023-06-23T01:02:36.820 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03f990d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f41158f0 con 0x7f03f4115200 2023-06-23T01:02:36.821 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03ebfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f4115200 0x7f03f410eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:36.822 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03ebfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f4115200 0x7f03f410eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:39702/0 (socket says 172.21.15.131:39702) 2023-06-23T01:02:36.822 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03ebfff700 1 -- 172.21.15.131:0/2181146980 learned_addr learned my addr 172.21.15.131:0/2181146980 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-06-23T01:02:36.822 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03ebfff700 1 -- 172.21.15.131:0/2181146980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f4115200 msgr2=0x7f03f410eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-06-23T01:02:36.822 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03ebfff700 1 -- 172.21.15.131:0/2181146980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f4115200 msgr2=0x7f03f410eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-06-23T01:02:36.822 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03ebfff700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f4115200 0x7f03f410eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:02:36.822 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03f2ffd700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03e0003580 0x7f03f4114cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:36.822 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.815+0000 7f03ebfff700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f4115200 0x7f03f410eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:02:36.823 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.816+0000 7f03f2ffd700 1 -- 172.21.15.131:0/2181146980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f4115200 msgr2=0x7f03f410eb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-23T01:02:36.824 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.816+0000 7f03f2ffd700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f4115200 0x7f03f410eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:36.824 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.816+0000 7f03f2ffd700 1 -- 172.21.15.131:0/2181146980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e4005040 con 0x7f03e0003580 2023-06-23T01:02:36.824 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.816+0000 7f03f2ffd700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03e0003580 0x7f03f4114cc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f03e4004fd0 tx=0x7f03e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:36.824 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.816+0000 7f03eb7fe700 1 -- 172.21.15.131:0/2181146980 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f03e401b070 con 0x7f03e0003580 2023-06-23T01:02:36.824 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.816+0000 7f03eb7fe700 1 -- 172.21.15.131:0/2181146980 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f03e4020a90 con 0x7f03e0003580 2023-06-23T01:02:36.824 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.816+0000 7f03eb7fe700 1 -- 172.21.15.131:0/2181146980 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f03e401fa00 con 0x7f03e0003580 2023-06-23T01:02:36.825 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.816+0000 7f03f990d700 1 -- 172.21.15.131:0/2181146980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f410f160 con 0x7f03e0003580 2023-06-23T01:02:36.825 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.817+0000 7f03f990d700 1 -- 172.21.15.131:0/2181146980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f410f670 con 0x7f03e0003580 2023-06-23T01:02:36.825 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.817+0000 7f03eb7fe700 1 -- 172.21.15.131:0/2181146980 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f03e401fc70 con 0x7f03e0003580 2023-06-23T01:02:36.825 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.818+0000 7f03eb7fe700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03d40367f0 0x7f03d4038ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:36.825 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.818+0000 7f03eb7fe700 1 -- 172.21.15.131:0/2181146980 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f03e404a7b0 con 0x7f03e0003580 2023-06-23T01:02:36.826 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.818+0000 7f03f990d700 1 -- 172.21.15.131:0/2181146980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f405f060 con 0x7f03e0003580 2023-06-23T01:02:36.826 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.818+0000 7f03ebfff700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03d40367f0 0x7f03d4038ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:36.826 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.818+0000 7f03ebfff700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03d40367f0 0x7f03d4038ca0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f03dc006fd0 tx=0x7f03dc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:36.826 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:36.824+0000 7f03eb7fe700 1 -- 172.21.15.131:0/2181146980 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f03e4030300 con 0x7f03e0003580 2023-06-23T01:02:37.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:36 smithi046 ceph-mon[84949]: Deploying daemon mon.b on smithi092 2023-06-23T01:02:37.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:36 smithi046 ceph-mon[84949]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:37.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:37.086 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.083+0000 7f03f990d700 1 -- 172.21.15.131:0/2181146980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03f41102a0 con 0x7f03e0003580 2023-06-23T01:02:37.086 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.083+0000 7f03eb7fe700 1 -- 172.21.15.131:0/2181146980 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1001 (secure 0 0 0) 0x7f03e4049020 con 0x7f03e0003580 2023-06-23T01:02:37.087 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T01:02:37.087 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":2,"fsid":"458642ee-1161-11ee-9b2b-001a4aab830c","modified":"2023-06-23T01:02:30.659063Z","created":"2023-06-23T01:00:55.582541Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-06-23T01:02:37.089 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.085+0000 7f03f990d700 1 -- 172.21.15.131:0/2181146980 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03d40367f0 msgr2=0x7f03d4038ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:37.089 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.085+0000 7f03f990d700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03d40367f0 0x7f03d4038ca0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f03dc006fd0 tx=0x7f03dc008040 comp rx=0 tx=0).stop 2023-06-23T01:02:37.089 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.085+0000 7f03f990d700 1 -- 172.21.15.131:0/2181146980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03e0003580 msgr2=0x7f03f4114cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:37.090 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.085+0000 7f03f990d700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03e0003580 0x7f03f4114cc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f03e4004fd0 tx=0x7f03e400b040 comp rx=0 tx=0).stop 2023-06-23T01:02:37.090 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.085+0000 7f03f990d700 1 -- 172.21.15.131:0/2181146980 shutdown_connections 2023-06-23T01:02:37.090 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.085+0000 7f03f990d700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03d40367f0 0x7f03d4038ca0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:37.091 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.085+0000 7f03f990d700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03e0003580 0x7f03f4114cc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:37.091 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.085+0000 7f03f990d700 1 --2- 172.21.15.131:0/2181146980 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f4115200 0x7f03f410eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:37.091 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.085+0000 7f03f990d700 1 -- 172.21.15.131:0/2181146980 >> 172.21.15.131:0/2181146980 conn(0x7f03f40f8300 msgr2=0x7f03f40fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:37.091 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.085+0000 7f03f990d700 1 -- 172.21.15.131:0/2181146980 shutdown_connections 2023-06-23T01:02:37.092 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:37.085+0000 7f03f990d700 1 -- 172.21.15.131:0/2181146980 wait complete. 2023-06-23T01:02:37.092 INFO:teuthology.orchestra.run.smithi131.stderr:dumped monmap epoch 2 2023-06-23T01:02:37.113 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:36 smithi131 ceph-mon[86495]: Deploying daemon mon.b on smithi092 2023-06-23T01:02:37.113 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:36 smithi131 ceph-mon[86495]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:37.113 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:37.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:37 smithi131 ceph-mon[86495]: from='client.? 172.21.15.131:0/2181146980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-23T01:02:38.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:37 smithi046 ceph-mon[84949]: from='client.? 172.21.15.131:0/2181146980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-23T01:02:38.550 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-06-23T01:02:38.551 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph mon dump -f json 2023-06-23T01:02:38.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 systemd[1]: Starting Ceph mon.b for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:02:38.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:38 smithi131 ceph-mon[86495]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:39.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:38 smithi046 ceph-mon[84949]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:39.132 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:02:39.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: set uid:gid to 167:167 (ceph:ceph) 2023-06-23T01:02:39.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: ceph version 17.2.6-719-g42ddc46e (42ddc46e6e6582d8e29a5a982a81886fe028be2c) quincy (stable), process ceph-mon, pid 7 2023-06-23T01:02:39.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: pidfile_write: ignore empty --pid-file 2023-06-23T01:02:39.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: load: jerasure load: lrc 2023-06-23T01:02:39.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: RocksDB version: 6.15.5 2023-06-23T01:02:39.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-06-23T01:02:39.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Compile date Jun 21 2023 2023-06-23T01:02:39.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: DB SUMMARY 2023-06-23T01:02:39.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: DB Session ID: SIABNM8KQLK5T7GD8VEP 2023-06-23T01:02:39.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: CURRENT file: CURRENT 2023-06-23T01:02:39.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: IDENTITY file: IDENTITY 2023-06-23T01:02:39.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2023-06-23T01:02:39.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-06-23T01:02:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-06-23T01:02:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.error_if_exists: 0 2023-06-23T01:02:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.create_if_missing: 0 2023-06-23T01:02:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.paranoid_checks: 1 2023-06-23T01:02:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-06-23T01:02:39.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.env: 0x55aa8377aa80 2023-06-23T01:02:39.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.fs: Posix File System 2023-06-23T01:02:39.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.info_log: 0x55aa859ffda0 2023-06-23T01:02:39.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_file_opening_threads: 16 2023-06-23T01:02:39.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.statistics: (nil) 2023-06-23T01:02:39.145 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.use_fsync: 0 2023-06-23T01:02:39.145 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_log_file_size: 0 2023-06-23T01:02:39.145 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-06-23T01:02:39.145 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.log_file_time_to_roll: 0 2023-06-23T01:02:39.146 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.keep_log_file_num: 1000 2023-06-23T01:02:39.146 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.recycle_log_file_num: 0 2023-06-23T01:02:39.146 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.allow_fallocate: 1 2023-06-23T01:02:39.146 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.allow_mmap_reads: 0 2023-06-23T01:02:39.147 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.allow_mmap_writes: 0 2023-06-23T01:02:39.147 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.use_direct_reads: 0 2023-06-23T01:02:39.147 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-06-23T01:02:39.147 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.create_missing_column_families: 0 2023-06-23T01:02:39.148 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.db_log_dir: 2023-06-23T01:02:39.148 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2023-06-23T01:02:39.148 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.table_cache_numshardbits: 6 2023-06-23T01:02:39.149 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.WAL_ttl_seconds: 0 2023-06-23T01:02:39.149 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.WAL_size_limit_MB: 0 2023-06-23T01:02:39.149 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-06-23T01:02:39.149 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-06-23T01:02:39.150 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.is_fd_close_on_exec: 1 2023-06-23T01:02:39.150 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.advise_random_on_open: 1 2023-06-23T01:02:39.150 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.db_write_buffer_size: 0 2023-06-23T01:02:39.150 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.write_buffer_manager: 0x55aa85ac48d0 2023-06-23T01:02:39.151 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-06-23T01:02:39.151 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-06-23T01:02:39.151 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-06-23T01:02:39.151 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.use_adaptive_mutex: 0 2023-06-23T01:02:39.152 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.rate_limiter: (nil) 2023-06-23T01:02:39.152 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-06-23T01:02:39.152 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.wal_recovery_mode: 2 2023-06-23T01:02:39.153 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.enable_thread_tracking: 0 2023-06-23T01:02:39.153 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.enable_pipelined_write: 0 2023-06-23T01:02:39.153 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.unordered_write: 0 2023-06-23T01:02:39.153 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-06-23T01:02:39.153 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-06-23T01:02:39.154 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-06-23T01:02:39.154 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-06-23T01:02:39.154 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.row_cache: None 2023-06-23T01:02:39.154 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.wal_filter: None 2023-06-23T01:02:39.154 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-06-23T01:02:39.154 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.allow_ingest_behind: 0 2023-06-23T01:02:39.154 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.preserve_deletes: 0 2023-06-23T01:02:39.154 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.two_write_queues: 0 2023-06-23T01:02:39.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.manual_wal_flush: 0 2023-06-23T01:02:39.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.atomic_flush: 0 2023-06-23T01:02:39.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-06-23T01:02:39.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.persist_stats_to_disk: 0 2023-06-23T01:02:39.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.write_dbid_to_manifest: 0 2023-06-23T01:02:39.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.log_readahead_size: 0 2023-06-23T01:02:39.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-06-23T01:02:39.155 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.best_efforts_recovery: 0 2023-06-23T01:02:39.156 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-06-23T01:02:39.156 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-06-23T01:02:39.156 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.allow_data_in_errors: 0 2023-06-23T01:02:39.156 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.db_host_id: __hostname__ 2023-06-23T01:02:39.156 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_background_jobs: 2 2023-06-23T01:02:39.156 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_background_compactions: -1 2023-06-23T01:02:39.157 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_subcompactions: 1 2023-06-23T01:02:39.157 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-06-23T01:02:39.157 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-06-23T01:02:39.157 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.delayed_write_rate : 16777216 2023-06-23T01:02:39.157 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_total_wal_size: 0 2023-06-23T01:02:39.157 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-06-23T01:02:39.157 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.stats_dump_period_sec: 600 2023-06-23T01:02:39.157 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.stats_persist_period_sec: 600 2023-06-23T01:02:39.158 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-06-23T01:02:39.158 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_open_files: -1 2023-06-23T01:02:39.158 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.bytes_per_sync: 0 2023-06-23T01:02:39.158 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.wal_bytes_per_sync: 0 2023-06-23T01:02:39.158 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.strict_bytes_per_sync: 0 2023-06-23T01:02:39.158 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_readahead_size: 0 2023-06-23T01:02:39.158 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_background_flushes: -1 2023-06-23T01:02:39.158 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Compression algorithms supported: 2023-06-23T01:02:39.159 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-06-23T01:02:39.159 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: kZSTD supported: 0 2023-06-23T01:02:39.159 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: kXpressCompression supported: 0 2023-06-23T01:02:39.159 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: kLZ4HCCompression supported: 1 2023-06-23T01:02:39.159 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: kZlibCompression supported: 1 2023-06-23T01:02:39.159 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: kSnappyCompression supported: 1 2023-06-23T01:02:39.159 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: kLZ4Compression supported: 1 2023-06-23T01:02:39.159 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: kBZip2Compression supported: 0 2023-06-23T01:02:39.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Fast CRC32 supported: Supported on x86 2023-06-23T01:02:39.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2023-06-23T01:02:39.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-06-23T01:02:39.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-06-23T01:02:39.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.merge_operator: 2023-06-23T01:02:39.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_filter: None 2023-06-23T01:02:39.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_filter_factory: None 2023-06-23T01:02:39.160 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.sst_partitioner_factory: None 2023-06-23T01:02:39.161 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.memtable_factory: SkipListFactory 2023-06-23T01:02:39.161 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.table_factory: BlockBasedTable 2023-06-23T01:02:39.161 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55aa859ced30) 2023-06-23T01:02:39.161 INFO:journalctl@ceph.mon.b.smithi092.stdout: cache_index_and_filter_blocks: 1 2023-06-23T01:02:39.161 INFO:journalctl@ceph.mon.b.smithi092.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-06-23T01:02:39.161 INFO:journalctl@ceph.mon.b.smithi092.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-06-23T01:02:39.161 INFO:journalctl@ceph.mon.b.smithi092.stdout: pin_top_level_index_and_filter: 1 2023-06-23T01:02:39.161 INFO:journalctl@ceph.mon.b.smithi092.stdout: index_type: 0 2023-06-23T01:02:39.162 INFO:journalctl@ceph.mon.b.smithi092.stdout: data_block_index_type: 0 2023-06-23T01:02:39.163 INFO:journalctl@ceph.mon.b.smithi092.stdout: index_shortening: 1 2023-06-23T01:02:39.163 INFO:journalctl@ceph.mon.b.smithi092.stdout: data_block_hash_table_util_ratio: 0.750000 2023-06-23T01:02:39.163 INFO:journalctl@ceph.mon.b.smithi092.stdout: hash_index_allow_collision: 1 2023-06-23T01:02:39.163 INFO:journalctl@ceph.mon.b.smithi092.stdout: checksum: 1 2023-06-23T01:02:39.163 INFO:journalctl@ceph.mon.b.smithi092.stdout: no_block_cache: 0 2023-06-23T01:02:39.163 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache: 0x55aa85a871f0 2023-06-23T01:02:39.164 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache_name: BinnedLRUCache 2023-06-23T01:02:39.164 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache_options: 2023-06-23T01:02:39.164 INFO:journalctl@ceph.mon.b.smithi092.stdout: capacity : 536870912 2023-06-23T01:02:39.164 INFO:journalctl@ceph.mon.b.smithi092.stdout: num_shard_bits : 4 2023-06-23T01:02:39.164 INFO:journalctl@ceph.mon.b.smithi092.stdout: strict_capacity_limit : 0 2023-06-23T01:02:39.164 INFO:journalctl@ceph.mon.b.smithi092.stdout: high_pri_pool_ratio: 0.000 2023-06-23T01:02:39.164 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_cache_compressed: (nil) 2023-06-23T01:02:39.165 INFO:journalctl@ceph.mon.b.smithi092.stdout: persistent_cache: (nil) 2023-06-23T01:02:39.165 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_size: 4096 2023-06-23T01:02:39.165 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_size_deviation: 10 2023-06-23T01:02:39.165 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_restart_interval: 16 2023-06-23T01:02:39.165 INFO:journalctl@ceph.mon.b.smithi092.stdout: index_block_restart_interval: 1 2023-06-23T01:02:39.165 INFO:journalctl@ceph.mon.b.smithi092.stdout: metadata_block_size: 4096 2023-06-23T01:02:39.165 INFO:journalctl@ceph.mon.b.smithi092.stdout: partition_filters: 0 2023-06-23T01:02:39.165 INFO:journalctl@ceph.mon.b.smithi092.stdout: use_delta_encoding: 1 2023-06-23T01:02:39.166 INFO:journalctl@ceph.mon.b.smithi092.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2023-06-23T01:02:39.166 INFO:journalctl@ceph.mon.b.smithi092.stdout: whole_key_filtering: 1 2023-06-23T01:02:39.166 INFO:journalctl@ceph.mon.b.smithi092.stdout: verify_compression: 0 2023-06-23T01:02:39.166 INFO:journalctl@ceph.mon.b.smithi092.stdout: read_amp_bytes_per_bit: 0 2023-06-23T01:02:39.166 INFO:journalctl@ceph.mon.b.smithi092.stdout: format_version: 4 2023-06-23T01:02:39.166 INFO:journalctl@ceph.mon.b.smithi092.stdout: enable_index_compression: 1 2023-06-23T01:02:39.166 INFO:journalctl@ceph.mon.b.smithi092.stdout: block_align: 0 2023-06-23T01:02:39.167 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.write_buffer_size: 33554432 2023-06-23T01:02:39.167 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_write_buffer_number: 2 2023-06-23T01:02:39.167 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compression: NoCompression 2023-06-23T01:02:39.167 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.bottommost_compression: Disabled 2023-06-23T01:02:39.167 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.prefix_extractor: nullptr 2023-06-23T01:02:39.167 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-06-23T01:02:39.167 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.num_levels: 7 2023-06-23T01:02:39.167 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-06-23T01:02:39.168 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-06-23T01:02:39.168 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-06-23T01:02:39.168 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-06-23T01:02:39.168 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-06-23T01:02:39.168 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-06-23T01:02:39.168 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-06-23T01:02:39.168 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-06-23T01:02:39.169 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-06-23T01:02:39.169 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-06-23T01:02:39.169 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compression_opts.window_bits: -14 2023-06-23T01:02:39.169 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compression_opts.level: 32767 2023-06-23T01:02:39.169 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compression_opts.strategy: 0 2023-06-23T01:02:39.169 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-06-23T01:02:39.169 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-06-23T01:02:39.169 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-06-23T01:02:39.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compression_opts.enabled: false 2023-06-23T01:02:39.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-06-23T01:02:39.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-06-23T01:02:39.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-06-23T01:02:39.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.target_file_size_base: 67108864 2023-06-23T01:02:39.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.target_file_size_multiplier: 1 2023-06-23T01:02:39.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-06-23T01:02:39.170 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-06-23T01:02:39.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-06-23T01:02:39.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-06-23T01:02:39.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-06-23T01:02:39.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-06-23T01:02:39.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-06-23T01:02:39.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-06-23T01:02:39.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-06-23T01:02:39.171 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-06-23T01:02:39.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-06-23T01:02:39.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-06-23T01:02:39.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.arena_block_size: 4194304 2023-06-23T01:02:39.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-06-23T01:02:39.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-06-23T01:02:39.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-06-23T01:02:39.172 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.disable_auto_compactions: 0 2023-06-23T01:02:39.173 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-06-23T01:02:39.173 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-06-23T01:02:39.173 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-06-23T01:02:39.173 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-06-23T01:02:39.173 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-06-23T01:02:39.173 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-06-23T01:02:39.173 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-06-23T01:02:39.173 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-06-23T01:02:39.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-06-23T01:02:39.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-06-23T01:02:39.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.table_properties_collectors: 2023-06-23T01:02:39.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.inplace_update_support: 0 2023-06-23T01:02:39.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.inplace_update_num_locks: 10000 2023-06-23T01:02:39.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-06-23T01:02:39.174 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-06-23T01:02:39.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.memtable_huge_page_size: 0 2023-06-23T01:02:39.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.bloom_locality: 0 2023-06-23T01:02:39.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.max_successive_merges: 0 2023-06-23T01:02:39.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.optimize_filters_for_hits: 0 2023-06-23T01:02:39.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.paranoid_file_checks: 0 2023-06-23T01:02:39.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.force_consistency_checks: 1 2023-06-23T01:02:39.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.report_bg_io_stats: 0 2023-06-23T01:02:39.175 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.ttl: 2592000 2023-06-23T01:02:39.176 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.periodic_compaction_seconds: 0 2023-06-23T01:02:39.176 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.enable_blob_files: false 2023-06-23T01:02:39.176 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.min_blob_size: 0 2023-06-23T01:02:39.176 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.blob_file_size: 268435456 2023-06-23T01:02:39.176 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.blob_compression_type: NoCompression 2023-06-23T01:02:39.176 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.enable_blob_garbage_collection: false 2023-06-23T01:02:39.176 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-06-23T01:02:39.176 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-06-23T01:02:39.177 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2023-06-23T01:02:39.177 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2023-06-23T01:02:39.177 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687482158868608, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-06-23T01:02:39.177 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 bash[85606]: 5be273638372af6159059081d919b4c9bc4c178986a96d4d6842631b0afc48b0 2023-06-23T01:02:39.177 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2023-06-23T01:02:39.177 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-06-23T01:02:39.177 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687482158868876, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1687482158, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "5d4a7490-b932-40b0-a550-c74d5a9750b7", "db_session_id": "SIABNM8KQLK5T7GD8VEP"}} 2023-06-23T01:02:39.178 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2023-06-23T01:02:39.178 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: EVENT_LOG_v1 {"time_micros": 1687482158869218, "job": 1, "event": "recovery_finished"} 2023-06-23T01:02:39.178 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-06-23T01:02:39.178 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55aa85a1b180 2023-06-23T01:02:39.178 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: DB pointer 0x55aa85a36000 2023-06-23T01:02:39.178 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-06-23T01:02:39.178 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: rocksdb: [db/db_impl/db_impl.cc:903] 2023-06-23T01:02:39.178 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** DB Stats ** 2023-06-23T01:02:39.179 INFO:journalctl@ceph.mon.b.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-23T01:02:39.179 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-06-23T01:02:39.179 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-06-23T01:02:39.179 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-06-23T01:02:39.179 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-06-23T01:02:39.179 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-06-23T01:02:39.179 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-06-23T01:02:39.179 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-06-23T01:02:39.180 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** Compaction Stats [default] ** 2023-06-23T01:02:39.180 INFO:journalctl@ceph.mon.b.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-23T01:02:39.180 INFO:journalctl@ceph.mon.b.smithi092.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-23T01:02:39.180 INFO:journalctl@ceph.mon.b.smithi092.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-23T01:02:39.180 INFO:journalctl@ceph.mon.b.smithi092.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-23T01:02:39.180 INFO:journalctl@ceph.mon.b.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-23T01:02:39.180 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-06-23T01:02:39.181 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** Compaction Stats [default] ** 2023-06-23T01:02:39.181 INFO:journalctl@ceph.mon.b.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-23T01:02:39.181 INFO:journalctl@ceph.mon.b.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-23T01:02:39.181 INFO:journalctl@ceph.mon.b.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-23T01:02:39.181 INFO:journalctl@ceph.mon.b.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-23T01:02:39.181 INFO:journalctl@ceph.mon.b.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-06-23T01:02:39.181 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-06-23T01:02:39.182 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-06-23T01:02:39.182 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-06-23T01:02:39.182 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2023-06-23T01:02:39.182 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-23T01:02:39.182 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-23T01:02:39.183 INFO:journalctl@ceph.mon.b.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-06-23T01:02:39.183 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-06-23T01:02:39.183 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2023-06-23T01:02:39.183 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-06-23T01:02:39.183 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** Compaction Stats [default] ** 2023-06-23T01:02:39.184 INFO:journalctl@ceph.mon.b.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-23T01:02:39.184 INFO:journalctl@ceph.mon.b.smithi092.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-23T01:02:39.184 INFO:journalctl@ceph.mon.b.smithi092.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-23T01:02:39.184 INFO:journalctl@ceph.mon.b.smithi092.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-23T01:02:39.184 INFO:journalctl@ceph.mon.b.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-06-23T01:02:39.184 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-06-23T01:02:39.184 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** Compaction Stats [default] ** 2023-06-23T01:02:39.185 INFO:journalctl@ceph.mon.b.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-06-23T01:02:39.185 INFO:journalctl@ceph.mon.b.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-06-23T01:02:39.185 INFO:journalctl@ceph.mon.b.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.4 0.00 0.00 1 0.000 0 0 2023-06-23T01:02:39.185 INFO:journalctl@ceph.mon.b.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-06-23T01:02:39.185 INFO:journalctl@ceph.mon.b.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-06-23T01:02:39.186 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-06-23T01:02:39.186 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-06-23T01:02:39.186 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-06-23T01:02:39.186 INFO:journalctl@ceph.mon.b.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2023-06-23T01:02:39.186 INFO:journalctl@ceph.mon.b.smithi092.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-23T01:02:39.187 INFO:journalctl@ceph.mon.b.smithi092.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-06-23T01:02:39.187 INFO:journalctl@ceph.mon.b.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-06-23T01:02:39.187 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-06-23T01:02:39.187 INFO:journalctl@ceph.mon.b.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2023-06-23T01:02:39.187 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-06-23T01:02:39.187 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: using public_addr v2:172.21.15.92:0/0 -> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-06-23T01:02:39.187 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: starting mon.b rank -1 at public addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] at bind addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:02:39.187 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(???) e0 preinit fsid 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:02:39.188 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).mds e1 new map 2023-06-23T01:02:39.188 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).mds e1 print_map 2023-06-23T01:02:39.188 INFO:journalctl@ceph.mon.b.smithi092.stdout: e1 2023-06-23T01:02:39.188 INFO:journalctl@ceph.mon.b.smithi092.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-06-23T01:02:39.188 INFO:journalctl@ceph.mon.b.smithi092.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-06-23T01:02:39.188 INFO:journalctl@ceph.mon.b.smithi092.stdout: legacy client fscid: -1 2023-06-23T01:02:39.188 INFO:journalctl@ceph.mon.b.smithi092.stdout: 2023-06-23T01:02:39.189 INFO:journalctl@ceph.mon.b.smithi092.stdout: No filesystems configured 2023-06-23T01:02:39.189 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-06-23T01:02:39.189 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-06-23T01:02:39.189 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-06-23T01:02:39.189 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-06-23T01:02:39.189 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-06-23T01:02:39.190 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-06-23T01:02:39.190 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-06-23T01:02:39.190 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-06-23T01:02:39.190 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-06-23T01:02:39.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-06-23T01:02:39.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: Added host smithi131 2023-06-23T01:02:39.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:39.191 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.192 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:39.192 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.192 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-06-23T01:02:39.193 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:39.193 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/4114537337' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-06-23T01:02:39.193 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.193 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:02:39.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:39.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:39.194 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: Updating smithi131:/etc/ceph/ceph.conf 2023-06-23T01:02:39.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: Updating smithi131:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:39.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/4114537337' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-06-23T01:02:39.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: osdmap e4: 0 total, 0 up, 0 in 2023-06-23T01:02:39.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2023-06-23T01:02:39.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: Updating smithi131:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.client.admin.keyring 2023-06-23T01:02:39.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:39.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi092:172.21.15.92=b;smithi131:172.21.15.131=c", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:39.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: Saving service mon spec with placement smithi046:172.21.15.46=a;smithi092:172.21.15.92=b;smithi131:172.21.15.131=c;count:3 2023-06-23T01:02:39.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.196 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:39.197 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:39.197 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:39.197 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.197 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:39.197 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:39.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: Deploying daemon mon.c on smithi131 2023-06-23T01:02:39.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:39.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:02:39.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.a calling monitor election 2023-06-23T01:02:39.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:39.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:39.198 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:39.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:39.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.c calling monitor election 2023-06-23T01:02:39.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:39.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:39.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:39.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:39.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-23T01:02:39.199 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: monmap e2: 2 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],c=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} removed_ranks: {} 2023-06-23T01:02:39.200 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: fsmap 2023-06-23T01:02:39.200 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: osdmap e4: 0 total, 0 up, 0 in 2023-06-23T01:02:39.200 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mgrmap e13: x(active, since 39s) 2023-06-23T01:02:39.200 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: overall HEALTH_OK 2023-06-23T01:02:39.200 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.200 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.200 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:39.201 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:39.201 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:39.201 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: Deploying daemon mon.b on smithi092 2023-06-23T01:02:39.201 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:39.201 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:39.201 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: from='client.? 172.21.15.131:0/2181146980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-23T01:02:39.202 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:39.202 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-06-23T01:02:39.202 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 ceph-mon[85670]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-06-23T01:02:39.202 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:38 smithi092 systemd[1]: Started Ceph mon.b for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:02:44.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:02:44.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: mon.a calling monitor election 2023-06-23T01:02:44.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: mon.c calling monitor election 2023-06-23T01:02:44.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-23T01:02:44.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: monmap e3: 3 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} removed_ranks: {} 2023-06-23T01:02:44.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: fsmap 2023-06-23T01:02:44.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: osdmap e4: 0 total, 0 up, 0 in 2023-06-23T01:02:44.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: mgrmap e13: x(active, since 48s) 2023-06-23T01:02:44.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-06-23T01:02:44.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-06-23T01:02:44.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-06-23T01:02:44.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: mon.b (rank 2) addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] is down (out of quorum) 2023-06-23T01:02:44.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:44.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:44.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:02:44.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: mon.a calling monitor election 2023-06-23T01:02:44.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: mon.c calling monitor election 2023-06-23T01:02:44.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-23T01:02:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: monmap e3: 3 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} removed_ranks: {} 2023-06-23T01:02:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: fsmap 2023-06-23T01:02:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: osdmap e4: 0 total, 0 up, 0 in 2023-06-23T01:02:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: mgrmap e13: x(active, since 48s) 2023-06-23T01:02:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-06-23T01:02:44.321 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-06-23T01:02:44.321 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-06-23T01:02:44.321 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: mon.b (rank 2) addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] is down (out of quorum) 2023-06-23T01:02:44.321 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:44.321 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:44.593 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.590+0000 7fcf3d699700 1 -- 172.21.15.131:0/1082438757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20003580 msgr2=0x7fcf20005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:44.593 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.590+0000 7fcf3d699700 1 --2- 172.21.15.131:0/1082438757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20003580 0x7fcf20005a00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcf380fc6b0 tx=0x7fcf28002040 comp rx=0 tx=0).stop 2023-06-23T01:02:44.594 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.590+0000 7fcf3d699700 1 -- 172.21.15.131:0/1082438757 shutdown_connections 2023-06-23T01:02:44.594 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.590+0000 7fcf3d699700 1 --2- 172.21.15.131:0/1082438757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20003580 0x7fcf20005a00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:44.594 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.590+0000 7fcf3d699700 1 -- 172.21.15.131:0/1082438757 >> 172.21.15.131:0/1082438757 conn(0x7fcf380f8340 msgr2=0x7fcf380fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:44.594 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.590+0000 7fcf3d699700 1 -- 172.21.15.131:0/1082438757 shutdown_connections 2023-06-23T01:02:44.594 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.590+0000 7fcf3d699700 1 -- 172.21.15.131:0/1082438757 wait complete. 2023-06-23T01:02:44.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf3d699700 1 Processor -- start 2023-06-23T01:02:44.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf3d699700 1 -- start start 2023-06-23T01:02:44.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf3d699700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20003580 0x7fcf38112890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:44.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf3d699700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf38112dd0 0x7fcf3810c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:44.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf3d699700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf3810cc90 0x7fcf3810d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:44.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf3d699700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38113420 con 0x7fcf20003580 2023-06-23T01:02:44.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf3d699700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38119690 con 0x7fcf38112dd0 2023-06-23T01:02:44.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf3d699700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf381197d0 con 0x7fcf3810cc90 2023-06-23T01:02:44.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf377fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf3810cc90 0x7fcf3810d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:44.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf377fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf3810cc90 0x7fcf3810d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48408/0 (socket says 172.21.15.131:48408) 2023-06-23T01:02:44.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf377fe700 1 -- 172.21.15.131:0/1014815548 learned_addr learned my addr 172.21.15.131:0/1014815548 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-06-23T01:02:44.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf367fc700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf38112dd0 0x7fcf3810c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:44.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.591+0000 7fcf377fe700 1 -- 172.21.15.131:0/1014815548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf3810cc90 msgr2=0x7fcf3810d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-06-23T01:02:44.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf377fe700 1 -- 172.21.15.131:0/1014815548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf3810cc90 msgr2=0x7fcf3810d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-06-23T01:02:44.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf377fe700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf3810cc90 0x7fcf3810d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:02:44.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf36ffd700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20003580 0x7fcf38112890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:44.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf377fe700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf3810cc90 0x7fcf3810d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:02:44.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf367fc700 1 -- 172.21.15.131:0/1014815548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf38112dd0 msgr2=0x7fcf3810c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-06-23T01:02:44.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf367fc700 1 -- 172.21.15.131:0/1014815548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf38112dd0 msgr2=0x7fcf3810c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-06-23T01:02:44.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf367fc700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf38112dd0 0x7fcf3810c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:02:44.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf367fc700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf38112dd0 0x7fcf3810c750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:02:44.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf36ffd700 1 -- 172.21.15.131:0/1014815548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf3810cc90 msgr2=0x7fcf3810d130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-23T01:02:44.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf36ffd700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf3810cc90 0x7fcf3810d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:44.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf36ffd700 1 -- 172.21.15.131:0/1014815548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf38112dd0 msgr2=0x7fcf3810c750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-06-23T01:02:44.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf36ffd700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf38112dd0 0x7fcf3810c750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:44.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf36ffd700 1 -- 172.21.15.131:0/1014815548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf28005040 con 0x7fcf20003580 2023-06-23T01:02:44.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf36ffd700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20003580 0x7fcf38112890 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcf28004830 tx=0x7fcf28002040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:44.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf1f7fe700 1 -- 172.21.15.131:0/1014815548 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf280093f0 con 0x7fcf20003580 2023-06-23T01:02:44.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf1f7fe700 1 -- 172.21.15.131:0/1014815548 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcf2800f040 con 0x7fcf20003580 2023-06-23T01:02:44.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf3d699700 1 -- 172.21.15.131:0/1014815548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf38119a50 con 0x7fcf20003580 2023-06-23T01:02:44.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf1f7fe700 1 -- 172.21.15.131:0/1014815548 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2802bea0 con 0x7fcf20003580 2023-06-23T01:02:44.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.592+0000 7fcf3d699700 1 -- 172.21.15.131:0/1014815548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf38119fc0 con 0x7fcf20003580 2023-06-23T01:02:44.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.593+0000 7fcf1f7fe700 1 -- 172.21.15.131:0/1014815548 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fcf28009550 con 0x7fcf20003580 2023-06-23T01:02:44.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.593+0000 7fcf3d699700 1 -- 172.21.15.131:0/1014815548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3805f060 con 0x7fcf20003580 2023-06-23T01:02:44.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.593+0000 7fcf1f7fe700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcf24036ad0 0x7fcf24038f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:44.602 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.594+0000 7fcf1f7fe700 1 -- 172.21.15.131:0/1014815548 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcf2804a7d0 con 0x7fcf20003580 2023-06-23T01:02:44.602 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.594+0000 7fcf367fc700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcf24036ad0 0x7fcf24038f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:44.602 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.594+0000 7fcf367fc700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcf24036ad0 0x7fcf24038f80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c006f30 tx=0x7fcf2c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:44.602 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.599+0000 7fcf1f7fe700 1 -- 172.21.15.131:0/1014815548 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fcf28029340 con 0x7fcf20003580 2023-06-23T01:02:44.885 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.882+0000 7fcf3d699700 1 -- 172.21.15.131:0/1014815548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf3804b970 con 0x7fcf20003580 2023-06-23T01:02:44.886 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.883+0000 7fcf1f7fe700 1 -- 172.21.15.131:0/1014815548 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1265 (secure 0 0 0) 0x7fcf28029340 con 0x7fcf20003580 2023-06-23T01:02:44.886 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-06-23T01:02:44.886 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":3,"fsid":"458642ee-1161-11ee-9b2b-001a4aab830c","modified":"2023-06-23T01:02:38.890596Z","created":"2023-06-23T01:00:55.582541Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3300","nonce":0},{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-06-23T01:02:44.888 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 -- 172.21.15.131:0/1014815548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcf24036ad0 msgr2=0x7fcf24038f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:44.888 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcf24036ad0 0x7fcf24038f80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c006f30 tx=0x7fcf2c008040 comp rx=0 tx=0).stop 2023-06-23T01:02:44.888 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 -- 172.21.15.131:0/1014815548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20003580 msgr2=0x7fcf38112890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:44.888 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20003580 0x7fcf38112890 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcf28004830 tx=0x7fcf28002040 comp rx=0 tx=0).stop 2023-06-23T01:02:44.889 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 -- 172.21.15.131:0/1014815548 shutdown_connections 2023-06-23T01:02:44.889 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fcf3810cc90 0x7fcf3810d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:44.889 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20003580 0x7fcf38112890 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:44.889 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fcf24036ad0 0x7fcf24038f80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:44.889 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 --2- 172.21.15.131:0/1014815548 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcf38112dd0 0x7fcf3810c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:44.889 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 -- 172.21.15.131:0/1014815548 >> 172.21.15.131:0/1014815548 conn(0x7fcf380f8340 msgr2=0x7fcf38100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:44.889 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 -- 172.21.15.131:0/1014815548 shutdown_connections 2023-06-23T01:02:44.890 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:44.885+0000 7fcf3d699700 1 -- 172.21.15.131:0/1014815548 wait complete. 2023-06-23T01:02:44.890 INFO:teuthology.orchestra.run.smithi131.stderr:dumped monmap epoch 3 2023-06-23T01:02:45.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:44 smithi131 ceph-mon[86495]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:45.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:45.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:45.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:45.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:44 smithi131 ceph-mon[86495]: from='client.? 172.21.15.131:0/1014815548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-23T01:02:45.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:45.267 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-06-23T01:02:45.268 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph config generate-minimal-conf 2023-06-23T01:02:45.298 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:44 smithi046 ceph-mon[84949]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:45.299 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:45.299 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:45.299 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:45.299 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:44 smithi046 ceph-mon[84949]: from='client.? 172.21.15.131:0/1014815548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-23T01:02:45.299 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:45.878 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:02:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:02:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: mon.a calling monitor election 2023-06-23T01:02:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: mon.c calling monitor election 2023-06-23T01:02:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:02:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:46.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:46.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:46.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:46.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:46.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:46.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:46.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-06-23T01:02:46.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: monmap e3: 3 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} removed_ranks: {} 2023-06-23T01:02:46.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: fsmap 2023-06-23T01:02:46.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: osdmap e4: 0 total, 0 up, 0 in 2023-06-23T01:02:46.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: mgrmap e13: x(active, since 48s) 2023-06-23T01:02:46.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-06-23T01:02:46.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-06-23T01:02:46.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-06-23T01:02:46.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: mon.b (rank 2) addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] is down (out of quorum) 2023-06-23T01:02:46.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:46.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:46.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:46.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:46.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:46.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:46.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='client.? 172.21.15.131:0/1014815548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-06-23T01:02:46.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:47.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.049+0000 7f1fb86d7700 1 -- 172.21.15.46:0/1567166897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01026c0 msgr2=0x7f1fb0102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.049+0000 7f1fb86d7700 1 --2- 172.21.15.46:0/1567166897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01026c0 0x7f1fb0102a90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0048c0 tx=0x7f1fac009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.049+0000 7f1fb86d7700 1 -- 172.21.15.46:0/1567166897 shutdown_connections 2023-06-23T01:02:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.049+0000 7f1fb86d7700 1 --2- 172.21.15.46:0/1567166897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01026c0 0x7f1fb0102a90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.049+0000 7f1fb86d7700 1 -- 172.21.15.46:0/1567166897 >> 172.21.15.46:0/1567166897 conn(0x7f1fb0071b50 msgr2=0x7f1fb0071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.049+0000 7f1fb86d7700 1 -- 172.21.15.46:0/1567166897 shutdown_connections 2023-06-23T01:02:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.049+0000 7f1fb86d7700 1 -- 172.21.15.46:0/1567166897 wait complete. 2023-06-23T01:02:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.049+0000 7f1fb86d7700 1 Processor -- start 2023-06-23T01:02:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.049+0000 7f1fb86d7700 1 -- start start 2023-06-23T01:02:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb86d7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01026c0 0x7f1fb0112b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb86d7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1fb01130a0 0x7f1fb010ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb86d7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fb010cf60 0x7f1fb010d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb86d7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb01136f0 con 0x7f1fb01026c0 2023-06-23T01:02:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb86d7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb0119960 con 0x7f1fb010cf60 2023-06-23T01:02:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb86d7700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb0119aa0 con 0x7f1fb01130a0 2023-06-23T01:02:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb6473700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01026c0 0x7f1fb0112b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb6473700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01026c0 0x7f1fb0112b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55946/0 (socket says 172.21.15.46:55946) 2023-06-23T01:02:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb6473700 1 -- 172.21.15.46:0/4024941601 learned_addr learned my addr 172.21.15.46:0/4024941601 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb5c72700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1fb01130a0 0x7f1fb010ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb6473700 1 -- 172.21.15.46:0/4024941601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1fb01130a0 msgr2=0x7f1fb010ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb6473700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1fb01130a0 0x7f1fb010ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb6473700 1 -- 172.21.15.46:0/4024941601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fb010cf60 msgr2=0x7f1fb010d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb6c74700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fb010cf60 0x7f1fb010d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb6473700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fb010cf60 0x7f1fb010d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb6473700 1 -- 172.21.15.46:0/4024941601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac005040 con 0x7f1fb01026c0 2023-06-23T01:02:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb6c74700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fb010cf60 0x7f1fb010d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:02:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fb6473700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01026c0 0x7f1fb0112b60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1fac009000 tx=0x7f1fac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.050+0000 7f1fa6ffd700 1 -- 172.21.15.46:0/4024941601 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac00e040 con 0x7f1fb01026c0 2023-06-23T01:02:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.051+0000 7f1fa6ffd700 1 -- 172.21.15.46:0/4024941601 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1fac0097f0 con 0x7f1fb01026c0 2023-06-23T01:02:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.051+0000 7f1fa6ffd700 1 -- 172.21.15.46:0/4024941601 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac013d10 con 0x7f1fb01026c0 2023-06-23T01:02:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.051+0000 7f1fb86d7700 1 -- 172.21.15.46:0/4024941601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb0119d20 con 0x7f1fb01026c0 2023-06-23T01:02:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.051+0000 7f1fb86d7700 1 -- 172.21.15.46:0/4024941601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb011a160 con 0x7f1fb01026c0 2023-06-23T01:02:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.051+0000 7f1fa6ffd700 1 -- 172.21.15.46:0/4024941601 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f1fac013420 con 0x7f1fb01026c0 2023-06-23T01:02:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.051+0000 7f1fb86d7700 1 -- 172.21.15.46:0/4024941601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb005f060 con 0x7f1fb01026c0 2023-06-23T01:02:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.052+0000 7f1fa6ffd700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1f9c036b20 0x7f1f9c038fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.052+0000 7f1fa6ffd700 1 -- 172.21.15.46:0/4024941601 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1fac049650 con 0x7f1fb01026c0 2023-06-23T01:02:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.052+0000 7f1fb5c72700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1f9c036b20 0x7f1f9c038fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.052+0000 7f1fb5c72700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1f9c036b20 0x7f1f9c038fd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009ac0 tx=0x7f1fa0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.056+0000 7f1fa6ffd700 1 -- 172.21.15.46:0/4024941601 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f1fac023300 con 0x7f1fb01026c0 2023-06-23T01:02:47.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: mon.b calling monitor election 2023-06-23T01:02:47.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: Updating smithi046:/etc/ceph/ceph.conf 2023-06-23T01:02:47.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: Updating smithi092:/etc/ceph/ceph.conf 2023-06-23T01:02:47.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: Updating smithi131:/etc/ceph/ceph.conf 2023-06-23T01:02:47.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:47.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: mon.b calling monitor election 2023-06-23T01:02:47.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: mon.c calling monitor election 2023-06-23T01:02:47.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: mon.a calling monitor election 2023-06-23T01:02:47.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-06-23T01:02:47.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: monmap e3: 3 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} removed_ranks: {} 2023-06-23T01:02:47.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: fsmap 2023-06-23T01:02:47.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: osdmap e4: 0 total, 0 up, 0 in 2023-06-23T01:02:47.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: mgrmap e13: x(active, since 50s) 2023-06-23T01:02:47.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-06-23T01:02:47.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: Cluster is now healthy 2023-06-23T01:02:47.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: overall HEALTH_OK 2023-06-23T01:02:47.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:47.145 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-23T01:02:47.145 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:47.148 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: mon.b calling monitor election 2023-06-23T01:02:47.148 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: Updating smithi046:/etc/ceph/ceph.conf 2023-06-23T01:02:47.149 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: Updating smithi092:/etc/ceph/ceph.conf 2023-06-23T01:02:47.149 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: Updating smithi131:/etc/ceph/ceph.conf 2023-06-23T01:02:47.149 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:47.149 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: mon.b calling monitor election 2023-06-23T01:02:47.149 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: mon.c calling monitor election 2023-06-23T01:02:47.149 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: mon.a calling monitor election 2023-06-23T01:02:47.149 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-06-23T01:02:47.150 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: monmap e3: 3 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} removed_ranks: {} 2023-06-23T01:02:47.150 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: fsmap 2023-06-23T01:02:47.150 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: osdmap e4: 0 total, 0 up, 0 in 2023-06-23T01:02:47.150 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: mgrmap e13: x(active, since 50s) 2023-06-23T01:02:47.150 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-06-23T01:02:47.150 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: Cluster is now healthy 2023-06-23T01:02:47.150 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: overall HEALTH_OK 2023-06-23T01:02:47.150 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-23T01:02:47.152 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: mon.b calling monitor election 2023-06-23T01:02:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: Updating smithi046:/etc/ceph/ceph.conf 2023-06-23T01:02:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: Updating smithi092:/etc/ceph/ceph.conf 2023-06-23T01:02:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: Updating smithi131:/etc/ceph/ceph.conf 2023-06-23T01:02:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: mon.b calling monitor election 2023-06-23T01:02:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: mon.c calling monitor election 2023-06-23T01:02:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: mon.a calling monitor election 2023-06-23T01:02:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-06-23T01:02:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: monmap e3: 3 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],b=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0]} removed_ranks: {} 2023-06-23T01:02:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: fsmap 2023-06-23T01:02:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: osdmap e4: 0 total, 0 up, 0 in 2023-06-23T01:02:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: mgrmap e13: x(active, since 50s) 2023-06-23T01:02:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-06-23T01:02:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: Cluster is now healthy 2023-06-23T01:02:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: overall HEALTH_OK 2023-06-23T01:02:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:47.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:47.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-23T01:02:47.235 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:47.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.255+0000 7f1fb86d7700 1 -- 172.21.15.46:0/4024941601 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f1fb011a350 con 0x7f1fb01026c0 2023-06-23T01:02:47.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.256+0000 7f1fa6ffd700 1 -- 172.21.15.46:0/4024941601 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7f1fac011b60 con 0x7f1fb01026c0 2023-06-23T01:02:47.258 INFO:teuthology.orchestra.run.smithi046.stdout:# minimal ceph.conf for 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:02:47.259 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2023-06-23T01:02:47.259 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 458642ee-1161-11ee-9b2b-001a4aab830c 2023-06-23T01:02:47.259 INFO:teuthology.orchestra.run.smithi046.stdout: mon_host = [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] 2023-06-23T01:02:47.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 -- 172.21.15.46:0/4024941601 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1f9c036b20 msgr2=0x7f1f9c038fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:47.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1f9c036b20 0x7f1f9c038fd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009ac0 tx=0x7f1fa0008040 comp rx=0 tx=0).stop 2023-06-23T01:02:47.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 -- 172.21.15.46:0/4024941601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01026c0 msgr2=0x7f1fb0112b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:47.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01026c0 0x7f1fb0112b60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1fac009000 tx=0x7f1fac009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:47.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 -- 172.21.15.46:0/4024941601 shutdown_connections 2023-06-23T01:02:47.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1f9c036b20 0x7f1f9c038fd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:47.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1fb010cf60 0x7f1fb010d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:47.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb01026c0 0x7f1fb0112b60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:47.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 --2- 172.21.15.46:0/4024941601 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1fb01130a0 0x7f1fb010ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:47.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 -- 172.21.15.46:0/4024941601 >> 172.21.15.46:0/4024941601 conn(0x7f1fb0071b50 msgr2=0x7f1fb00f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:47.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 -- 172.21.15.46:0/4024941601 shutdown_connections 2023-06-23T01:02:47.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:47.258+0000 7f1fb86d7700 1 -- 172.21.15.46:0/4024941601 wait complete. 2023-06-23T01:02:47.925 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-06-23T01:02:47.925 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T01:02:47.926 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.conf 2023-06-23T01:02:47.994 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T01:02:47.994 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-06-23T01:02:48.059 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:02:48.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.conf 2023-06-23T01:02:48.089 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:02:48.089 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-06-23T01:02:48.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:47 smithi092 ceph-mon[85670]: Updating smithi046:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:48.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:47 smithi092 ceph-mon[85670]: Updating smithi092:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:48.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:47 smithi092 ceph-mon[85670]: Updating smithi131:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:48.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:47 smithi092 ceph-mon[85670]: Reconfiguring mon.a (unknown last config time)... 2023-06-23T01:02:48.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:47 smithi092 ceph-mon[85670]: Reconfiguring daemon mon.a on smithi046 2023-06-23T01:02:48.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:47 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:48.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:47 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/4024941601' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:48.155 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-06-23T01:02:48.156 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.conf 2023-06-23T01:02:48.183 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:47 smithi131 ceph-mon[86495]: Updating smithi046:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:48.184 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:47 smithi131 ceph-mon[86495]: Updating smithi092:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:48.184 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:47 smithi131 ceph-mon[86495]: Updating smithi131:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:48.184 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:47 smithi131 ceph-mon[86495]: Reconfiguring mon.a (unknown last config time)... 2023-06-23T01:02:48.185 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:47 smithi131 ceph-mon[86495]: Reconfiguring daemon mon.a on smithi046 2023-06-23T01:02:48.185 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:47 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:48.185 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:47 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/4024941601' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:48.189 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-06-23T01:02:48.190 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-06-23T01:02:48.257 INFO:tasks.cephadm:Adding mgr.x on smithi046 2023-06-23T01:02:48.258 INFO:tasks.cephadm:Adding mgr.y on smithi092 2023-06-23T01:02:48.258 INFO:tasks.cephadm:Adding mgr.z on smithi131 2023-06-23T01:02:48.258 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch apply mgr '3;smithi046=x;smithi092=y;smithi131=z' 2023-06-23T01:02:48.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:47 smithi046 ceph-mon[84949]: Updating smithi046:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:48.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:47 smithi046 ceph-mon[84949]: Updating smithi092:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:48.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:47 smithi046 ceph-mon[84949]: Updating smithi131:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/config/ceph.conf 2023-06-23T01:02:48.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:47 smithi046 ceph-mon[84949]: Reconfiguring mon.a (unknown last config time)... 2023-06-23T01:02:48.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:47 smithi046 ceph-mon[84949]: Reconfiguring daemon mon.a on smithi046 2023-06-23T01:02:48.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:47 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:02:48.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:47 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/4024941601' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:48.877 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:02:49.120 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:48 smithi131 ceph-mon[86495]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:49.120 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:48 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:49.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:48 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:49.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:48 smithi131 ceph-mon[86495]: Reconfiguring mon.b (monmap changed)... 2023-06-23T01:02:49.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:48 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:49.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:48 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-23T01:02:49.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:48 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:49.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:48 smithi131 ceph-mon[86495]: Reconfiguring daemon mon.b on smithi092 2023-06-23T01:02:49.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:48 smithi092 ceph-mon[85670]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:49.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:48 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:49.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:48 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:49.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:48 smithi092 ceph-mon[85670]: Reconfiguring mon.b (monmap changed)... 2023-06-23T01:02:49.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:48 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:49.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:48 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-23T01:02:49.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:48 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:49.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:48 smithi092 ceph-mon[85670]: Reconfiguring daemon mon.b on smithi092 2023-06-23T01:02:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:48 smithi046 ceph-mon[84949]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:48 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:48 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:48 smithi046 ceph-mon[84949]: Reconfiguring mon.b (monmap changed)... 2023-06-23T01:02:49.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:48 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:49.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:48 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-23T01:02:49.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:48 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:49.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:48 smithi046 ceph-mon[84949]: Reconfiguring daemon mon.b on smithi092 2023-06-23T01:02:50.309 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:50 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:50.309 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:50 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:50.310 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:50 smithi131 ceph-mon[86495]: Reconfiguring mon.c (monmap changed)... 2023-06-23T01:02:50.310 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:50 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:50.310 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:50 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-23T01:02:50.310 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:50 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:50.310 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:50 smithi131 ceph-mon[86495]: Reconfiguring daemon mon.c on smithi131 2023-06-23T01:02:50.310 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:50 smithi131 ceph-mon[86495]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:50.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:50 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:50.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:50 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:50.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:50 smithi046 ceph-mon[84949]: Reconfiguring mon.c (monmap changed)... 2023-06-23T01:02:50.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:50 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:50.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:50 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-23T01:02:50.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:50 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:50.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:50 smithi046 ceph-mon[84949]: Reconfiguring daemon mon.c on smithi131 2023-06-23T01:02:50.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:50 smithi046 ceph-mon[84949]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:50.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:50 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:50.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:50 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:50.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:50 smithi092 ceph-mon[85670]: Reconfiguring mon.c (monmap changed)... 2023-06-23T01:02:50.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:50 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-06-23T01:02:50.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:50 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-06-23T01:02:50.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:50 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:50.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:50 smithi092 ceph-mon[85670]: Reconfiguring daemon mon.c on smithi131 2023-06-23T01:02:50.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:50 smithi092 ceph-mon[85670]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:50.477 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.450+0000 7f6bbd901700 1 -- 172.21.15.131:0/3765822467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb80fcd50 msgr2=0x7f6bb80fd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:50.477 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.450+0000 7f6bbd901700 1 --2- 172.21.15.131:0/3765822467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb80fcd50 0x7f6bb80fd1a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0004a40 tx=0x7f6ba0009e30 comp rx=0 tx=0).stop 2023-06-23T01:02:50.478 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.450+0000 7f6bbd901700 1 -- 172.21.15.131:0/3765822467 shutdown_connections 2023-06-23T01:02:50.478 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.450+0000 7f6bbd901700 1 --2- 172.21.15.131:0/3765822467 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb80fbb40 0x7f6bb80fbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:50.478 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.450+0000 7f6bbd901700 1 --2- 172.21.15.131:0/3765822467 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bb80fd6e0 0x7f6bb8103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:50.478 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.450+0000 7f6bbd901700 1 --2- 172.21.15.131:0/3765822467 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb80fcd50 0x7f6bb80fd1a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:50.478 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.450+0000 7f6bbd901700 1 -- 172.21.15.131:0/3765822467 >> 172.21.15.131:0/3765822467 conn(0x7f6bb80f72c0 msgr2=0x7f6bb80f96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:50.478 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.450+0000 7f6bbd901700 1 -- 172.21.15.131:0/3765822467 shutdown_connections 2023-06-23T01:02:50.478 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.450+0000 7f6bbd901700 1 -- 172.21.15.131:0/3765822467 wait complete. 2023-06-23T01:02:50.479 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bbd901700 1 Processor -- start 2023-06-23T01:02:50.479 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bbd901700 1 -- start start 2023-06-23T01:02:50.479 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bbd901700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bb80fbb40 0x7f6bb810fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:50.479 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bbd901700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb80fcd50 0x7f6bb810ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:50.479 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bbd901700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb80fd6e0 0x7f6bb8117050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:50.479 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bbd901700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8117660 con 0x7f6bb80fbb40 2023-06-23T01:02:50.479 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bbd901700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb81177d0 con 0x7f6bb80fcd50 2023-06-23T01:02:50.479 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bbd901700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8117990 con 0x7f6bb80fd6e0 2023-06-23T01:02:50.480 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bb77fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb80fd6e0 0x7f6bb8117050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:50.480 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bb77fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb80fd6e0 0x7f6bb8117050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48434/0 (socket says 172.21.15.131:48434) 2023-06-23T01:02:50.480 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bb77fe700 1 -- 172.21.15.131:0/1993627987 learned_addr learned my addr 172.21.15.131:0/1993627987 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-06-23T01:02:50.480 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bb6ffd700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bb80fbb40 0x7f6bb810fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:50.480 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6baffff700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb80fcd50 0x7f6bb810ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:50.480 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bb77fe700 1 -- 172.21.15.131:0/1993627987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb80fcd50 msgr2=0x7f6bb810ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:50.480 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bb77fe700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb80fcd50 0x7f6bb810ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:50.481 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bb77fe700 1 -- 172.21.15.131:0/1993627987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bb80fbb40 msgr2=0x7f6bb810fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:50.481 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bb77fe700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bb80fbb40 0x7f6bb810fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:50.481 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bb77fe700 1 -- 172.21.15.131:0/1993627987 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba0005040 con 0x7f6bb80fd6e0 2023-06-23T01:02:50.481 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bb6ffd700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bb80fbb40 0x7f6bb810fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:02:50.481 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bb77fe700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb80fd6e0 0x7f6bb8117050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000ec30 tx=0x7f6bb000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:50.481 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6baf7fe700 1 -- 172.21.15.131:0/1993627987 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0007b30 con 0x7f6bb80fd6e0 2023-06-23T01:02:50.481 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6baf7fe700 1 -- 172.21.15.131:0/1993627987 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6bb000f040 con 0x7f6bb80fd6e0 2023-06-23T01:02:50.481 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6bbd901700 1 -- 172.21.15.131:0/1993627987 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb8117c70 con 0x7f6bb80fd6e0 2023-06-23T01:02:50.482 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.451+0000 7f6baf7fe700 1 -- 172.21.15.131:0/1993627987 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb000ca50 con 0x7f6bb80fd6e0 2023-06-23T01:02:50.482 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.452+0000 7f6bbd901700 1 -- 172.21.15.131:0/1993627987 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb8118160 con 0x7f6bb80fd6e0 2023-06-23T01:02:50.482 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.452+0000 7f6baf7fe700 1 -- 172.21.15.131:0/1993627987 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f6bb0014480 con 0x7f6bb80fd6e0 2023-06-23T01:02:50.482 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.453+0000 7f6bbd901700 1 -- 172.21.15.131:0/1993627987 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb8103100 con 0x7f6bb80fd6e0 2023-06-23T01:02:50.482 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.453+0000 7f6baf7fe700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6ba4036890 0x7f6ba4038d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:50.483 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.453+0000 7f6baf7fe700 1 -- 172.21.15.131:0/1993627987 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6bb0048970 con 0x7f6bb80fd6e0 2023-06-23T01:02:50.483 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.453+0000 7f6bb6ffd700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6ba4036890 0x7f6ba4038d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:50.483 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.453+0000 7f6bb6ffd700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6ba4036890 0x7f6ba4038d40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8009fd0 tx=0x7f6ba8009f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:50.483 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.458+0000 7f6baf7fe700 1 -- 172.21.15.131:0/1993627987 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f6bb0012af0 con 0x7f6bb80fd6e0 2023-06-23T01:02:50.659 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.656+0000 7f6bbd901700 1 -- 172.21.15.131:0/1993627987 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi092=y;smithi131=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bb802b2c0 con 0x7f6ba4036890 2023-06-23T01:02:50.666 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.663+0000 7f6baf7fe700 1 -- 172.21.15.131:0/1993627987 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6bb802b2c0 con 0x7f6ba4036890 2023-06-23T01:02:50.666 INFO:teuthology.orchestra.run.smithi131.stdout:Scheduled mgr update... 2023-06-23T01:02:50.668 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 -- 172.21.15.131:0/1993627987 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6ba4036890 msgr2=0x7f6ba4038d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:50.668 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6ba4036890 0x7f6ba4038d40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8009fd0 tx=0x7f6ba8009f00 comp rx=0 tx=0).stop 2023-06-23T01:02:50.668 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 -- 172.21.15.131:0/1993627987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb80fd6e0 msgr2=0x7f6bb8117050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:50.668 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb80fd6e0 0x7f6bb8117050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000ec30 tx=0x7f6bb000b040 comp rx=0 tx=0).stop 2023-06-23T01:02:50.669 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 -- 172.21.15.131:0/1993627987 shutdown_connections 2023-06-23T01:02:50.669 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6bb80fd6e0 0x7f6bb8117050 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:50.669 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bb80fbb40 0x7f6bb810fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:50.669 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6ba4036890 0x7f6ba4038d40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:50.669 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 --2- 172.21.15.131:0/1993627987 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6bb80fcd50 0x7f6bb810ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:50.669 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 -- 172.21.15.131:0/1993627987 >> 172.21.15.131:0/1993627987 conn(0x7f6bb80f72c0 msgr2=0x7f6bb8100280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:50.670 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 -- 172.21.15.131:0/1993627987 shutdown_connections 2023-06-23T01:02:50.670 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:02:50.665+0000 7f6bbd901700 1 -- 172.21.15.131:0/1993627987 wait complete. 2023-06-23T01:02:51.073 DEBUG:teuthology.orchestra.run.smithi092:mgr.y> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.y.service 2023-06-23T01:02:51.076 DEBUG:teuthology.orchestra.run.smithi131:mgr.z> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.z.service 2023-06-23T01:02:51.078 INFO:tasks.cephadm:Deploying OSDs... 2023-06-23T01:02:51.079 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-06-23T01:02:51.079 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T01:02:51.098 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T01:02:51.099 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2023-06-23T01:02:51.106 INFO:journalctl@ceph.mgr.z.smithi131.stdout:-- Logs begin at Fri 2023-06-23 00:44:02 UTC. -- 2023-06-23T01:02:51.107 INFO:journalctl@ceph.mgr.y.smithi092.stdout:-- Logs begin at Fri 2023-06-23 00:44:08 UTC. -- 2023-06-23T01:02:51.157 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T01:02:51.158 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:51.158 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 210851 Links: 1 2023-06-23T01:02:51.158 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:51.159 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:51.159 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-06-23 01:02:05.333776082 +0000 2023-06-23T01:02:51.159 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-06-23 00:52:32.361202902 +0000 2023-06-23T01:02:51.159 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-06-23 00:52:32.361202902 +0000 2023-06-23T01:02:51.159 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2023-06-23T01:02:51.160 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T01:02:51.224 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-06-23T01:02:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-06-23T01:02:51.225 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000132399 s, 3.9 MB/s 2023-06-23T01:02:51.226 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T01:02:51.285 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2023-06-23T01:02:51.342 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T01:02:51.343 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:51.343 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 212152 Links: 1 2023-06-23T01:02:51.343 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:51.343 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:51.343 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-06-23 01:02:05.333776082 +0000 2023-06-23T01:02:51.343 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-06-23 00:52:32.886197397 +0000 2023-06-23T01:02:51.343 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-06-23 00:52:32.886197397 +0000 2023-06-23T01:02:51.344 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2023-06-23T01:02:51.344 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T01:02:51.408 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-06-23T01:02:51.409 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-06-23T01:02:51.409 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000128755 s, 4.0 MB/s 2023-06-23T01:02:51.410 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T01:02:51.467 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2023-06-23T01:02:51.524 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T01:02:51.525 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:51.525 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 214256 Links: 1 2023-06-23T01:02:51.525 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:51.525 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:51.525 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-06-23 01:02:05.333776082 +0000 2023-06-23T01:02:51.525 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-06-23 00:52:33.437191619 +0000 2023-06-23T01:02:51.525 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-06-23 00:52:33.437191619 +0000 2023-06-23T01:02:51.526 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2023-06-23T01:02:51.526 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T01:02:51.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:51.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:51.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:51.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi092=y;smithi131=z", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:51.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: Saving service mgr spec with placement smithi046=x;smithi092=y;smithi131=z;count:3 2023-06-23T01:02:51.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:51.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:51.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:51.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-23T01:02:51.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-23T01:02:51.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-23T01:02:51.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:51.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:51 smithi046 ceph-mon[84949]: Deploying daemon mgr.z on smithi131 2023-06-23T01:02:51.589 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-06-23T01:02:51.590 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-06-23T01:02:51.590 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000120944 s, 4.2 MB/s 2023-06-23T01:02:51.591 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T01:02:51.648 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2023-06-23T01:02:51.704 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T01:02:51.704 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:51.704 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 6h/6d Inode: 212283 Links: 1 2023-06-23T01:02:51.704 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:51.705 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:51.705 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-06-23 01:02:05.333776082 +0000 2023-06-23T01:02:51.705 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-06-23 00:52:33.988185842 +0000 2023-06-23T01:02:51.705 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-06-23 00:52:33.988185842 +0000 2023-06-23T01:02:51.705 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2023-06-23T01:02:51.705 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T01:02:51.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:51.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:51.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:51.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi092=y;smithi131=z", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:51.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: Saving service mgr spec with placement smithi046=x;smithi092=y;smithi131=z;count:3 2023-06-23T01:02:51.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:51.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:51.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:51.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-23T01:02:51.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-23T01:02:51.734 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-23T01:02:51.734 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:51.734 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:51 smithi131 ceph-mon[86495]: Deploying daemon mgr.z on smithi131 2023-06-23T01:02:51.768 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-06-23T01:02:51.769 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-06-23T01:02:51.769 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.00011376 s, 4.5 MB/s 2023-06-23T01:02:51.770 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T01:02:51.826 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:02:51.827 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T01:02:51.844 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T01:02:51.844 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2023-06-23T01:02:51.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:51.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:51.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:51.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi092=y;smithi131=z", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:51.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: Saving service mgr spec with placement smithi046=x;smithi092=y;smithi131=z;count:3 2023-06-23T01:02:51.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:51.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:51.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:51.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:51.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-23T01:02:51.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-23T01:02:51.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-23T01:02:51.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:51.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:51 smithi092 ceph-mon[85670]: Deploying daemon mgr.z on smithi131 2023-06-23T01:02:51.901 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T01:02:51.902 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:51.902 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 208458 Links: 1 2023-06-23T01:02:51.902 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:51.902 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:51.902 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-06-23 01:02:38.283853063 +0000 2023-06-23T01:02:51.902 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-06-23 00:52:15.517864999 +0000 2023-06-23T01:02:51.903 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-06-23 00:52:15.517864999 +0000 2023-06-23T01:02:51.903 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-06-23T01:02:51.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T01:02:51.969 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-06-23T01:02:51.970 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-06-23T01:02:51.970 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000280493 s, 1.8 MB/s 2023-06-23T01:02:51.972 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T01:02:52.029 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2023-06-23T01:02:52.085 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T01:02:52.086 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:52.086 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 205633 Links: 1 2023-06-23T01:02:52.086 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:52.086 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:52.087 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-06-23 01:02:38.283853063 +0000 2023-06-23T01:02:52.087 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-06-23 00:52:16.034849384 +0000 2023-06-23T01:02:52.087 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-06-23 00:52:16.034849384 +0000 2023-06-23T01:02:52.087 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-06-23T01:02:52.087 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T01:02:52.151 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-06-23T01:02:52.152 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-06-23T01:02:52.152 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000242918 s, 2.1 MB/s 2023-06-23T01:02:52.153 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T01:02:52.210 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2023-06-23T01:02:52.266 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T01:02:52.266 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:52.266 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 210664 Links: 1 2023-06-23T01:02:52.266 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:52.266 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:52.267 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-06-23 01:02:38.283853063 +0000 2023-06-23T01:02:52.267 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-06-23 00:52:16.543834012 +0000 2023-06-23T01:02:52.267 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-06-23 00:52:16.543834012 +0000 2023-06-23T01:02:52.267 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-06-23T01:02:52.267 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T01:02:52.330 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-06-23T01:02:52.331 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-06-23T01:02:52.331 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000245002 s, 2.1 MB/s 2023-06-23T01:02:52.332 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T01:02:52.388 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2023-06-23T01:02:52.415 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:52 smithi131 systemd[1]: Starting Ceph mgr.z for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:02:52.443 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T01:02:52.444 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:52.444 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 208730 Links: 1 2023-06-23T01:02:52.444 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:52.444 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:52.444 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-06-23 01:02:38.283853063 +0000 2023-06-23T01:02:52.445 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-06-23 00:52:17.060818398 +0000 2023-06-23T01:02:52.445 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-06-23 00:52:17.060818398 +0000 2023-06-23T01:02:52.445 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-06-23T01:02:52.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T01:02:52.508 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-06-23T01:02:52.508 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-06-23T01:02:52.508 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000210206 s, 2.4 MB/s 2023-06-23T01:02:52.509 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T01:02:52.564 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-06-23T01:02:52.565 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T01:02:52.580 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T01:02:52.581 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_1 2023-06-23T01:02:52.637 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T01:02:52.637 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:52.637 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 213244 Links: 1 2023-06-23T01:02:52.638 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:52.638 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:52.638 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2023-06-23 01:02:52.123644976 +0000 2023-06-23T01:02:52.638 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2023-06-23 00:52:17.051602762 +0000 2023-06-23T01:02:52.638 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2023-06-23 00:52:17.051602762 +0000 2023-06-23T01:02:52.638 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2023-06-23T01:02:52.639 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T01:02:52.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:52 smithi092 ceph-mon[85670]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:52.701 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:52 smithi131 ceph-mon[86495]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:52.704 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2023-06-23T01:02:52.705 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2023-06-23T01:02:52.705 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000254247 s, 2.0 MB/s 2023-06-23T01:02:52.706 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T01:02:52.763 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_2 2023-06-23T01:02:52.818 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T01:02:52.818 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:52.819 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 213393 Links: 1 2023-06-23T01:02:52.819 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:52.819 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:52.819 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2023-06-23 01:02:52.123644976 +0000 2023-06-23T01:02:52.819 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2023-06-23 00:52:17.567591813 +0000 2023-06-23T01:02:52.820 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2023-06-23 00:52:17.567591813 +0000 2023-06-23T01:02:52.820 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2023-06-23T01:02:52.820 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T01:02:52.821 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:52 smithi046 ceph-mon[84949]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:52.883 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2023-06-23T01:02:52.883 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2023-06-23T01:02:52.883 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000231481 s, 2.2 MB/s 2023-06-23T01:02:52.884 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T01:02:52.941 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_3 2023-06-23T01:02:52.982 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:52 smithi131 ceph-mgr[89193]: -- 172.21.15.131:0/3877904567 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555d5e6c89c0 con 0x555d5e69f000 2023-06-23T01:02:52.982 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:52 smithi131 bash[89081]: 6d264864094724057b5c6a7ad4c2ce44b10e802aa20de240654a51389cb2cc80 2023-06-23T01:02:52.983 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:52 smithi131 systemd[1]: Started Ceph mgr.z for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:02:52.997 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T01:02:52.998 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:52.998 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 214418 Links: 1 2023-06-23T01:02:52.998 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:52.999 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:52.999 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2023-06-23 01:02:52.123644976 +0000 2023-06-23T01:02:52.999 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2023-06-23 00:52:18.084580844 +0000 2023-06-23T01:02:53.000 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2023-06-23 00:52:18.084580844 +0000 2023-06-23T01:02:53.000 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2023-06-23T01:02:53.000 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T01:02:53.061 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2023-06-23T01:02:53.062 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2023-06-23T01:02:53.062 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000250435 s, 2.0 MB/s 2023-06-23T01:02:53.063 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T01:02:53.120 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_4 2023-06-23T01:02:53.175 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T01:02:53.175 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:02:53.175 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 210898 Links: 1 2023-06-23T01:02:53.175 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:02:53.175 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:02:53.175 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2023-06-23 01:02:52.124644955 +0000 2023-06-23T01:02:53.176 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2023-06-23 00:52:18.627569324 +0000 2023-06-23T01:02:53.176 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2023-06-23 00:52:18.627569324 +0000 2023-06-23T01:02:53.176 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2023-06-23T01:02:53.176 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T01:02:53.238 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2023-06-23T01:02:53.238 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2023-06-23T01:02:53.238 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000225145 s, 2.3 MB/s 2023-06-23T01:02:53.239 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T01:02:53.295 INFO:tasks.cephadm:Deploying osd.0 on smithi046 with /dev/vg_nvme/lv_4... 2023-06-23T01:02:53.296 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-06-23T01:02:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:02:54.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:53 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:54.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:53 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:54.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:53 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:54.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:53 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-23T01:02:54.107 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:53 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-23T01:02:54.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:53 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-23T01:02:54.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:53 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:54.108 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:53 smithi092 ceph-mon[85670]: Deploying daemon mgr.y on smithi092 2023-06-23T01:02:54.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:53 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:54.115 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:53 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:54.115 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:53 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:54.115 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:53 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-23T01:02:54.115 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:53 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-23T01:02:54.115 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:53 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-23T01:02:54.115 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:53 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:54.115 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:53 smithi046 ceph-mon[84949]: Deploying daemon mgr.y on smithi092 2023-06-23T01:02:54.120 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:53 smithi131 conmon[89122]: 2023-06-23T01:02:53.960+0000 7f566ce841c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:02:54.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:53 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:54.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:53 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:54.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:53 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:54.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:53 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-23T01:02:54.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:53 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-06-23T01:02:54.121 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:53 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-23T01:02:54.122 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:53 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:54.122 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:53 smithi131 ceph-mon[86495]: Deploying daemon mgr.y on smithi092 2023-06-23T01:02:54.389 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:54 smithi092 systemd[1]: Starting Ceph mgr.y for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:02:54.423 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:54 smithi131 conmon[89122]: 2023-06-23T01:02:54.117+0000 7f566ce841c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:02:54.423 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:54 smithi131 conmon[89122]: 2023-06-23T01:02:54.206+0000 7f566ce841c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:02:54.424 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:54 smithi131 conmon[89122]: 2023-06-23T01:02:54.420+0000 7f566ce841c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:02:54.731 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:54 smithi131 conmon[89122]: 2023-06-23T01:02:54.622+0000 7f566ce841c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:02:55.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:54 smithi092 ceph-mon[85670]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:55.140 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:54 smithi092 ceph-mgr[87374]: -- 172.21.15.92:0/842588479 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d1466389c0 con 0x55d14660e800 2023-06-23T01:02:55.140 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:54 smithi092 bash[87310]: c8f96f7e267db363c92b5f7cea9e46acc602d1bca250c1d168bc440f83ebbff1 2023-06-23T01:02:55.140 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:54 smithi092 systemd[1]: Started Ceph mgr.y for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:02:55.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:54 smithi131 ceph-mon[86495]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:55.232 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:54 smithi131 conmon[89122]: 2023-06-23T01:02:54.781+0000 7f566ce841c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:02:55.233 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:54 smithi131 conmon[89122]: 2023-06-23T01:02:54.861+0000 7f566ce841c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:02:55.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:54 smithi046 ceph-mon[84949]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:55.403 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-06-23T01:02:55.403 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-06-23T01:02:55.404 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2023-06-23T01:02:55.404 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2023-06-23T01:02:55.404 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00852355 s, 1.2 GB/s 2023-06-23T01:02:55.404 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2023-06-23T01:02:55.731 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:55 smithi131 conmon[89122]: 2023-06-23T01:02:55.470+0000 7f566ce841c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:02:55.739 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi046:vg_nvme/lv_4 2023-06-23T01:02:56.314 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:56 smithi092 conmon[87350]: 2023-06-23T01:02:56.144+0000 7f190e3f41c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:02:56.314 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:56.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:56 smithi092 ceph-mon[85670]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:56.315 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:56.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:56 smithi046 ceph-mon[84949]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:56.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:02:56.482 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:55 smithi131 conmon[89122]: 2023-06-23T01:02:55.998+0000 7f566ce841c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:02:56.482 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:56 smithi131 conmon[89122]: 2023-06-23T01:02:56.100+0000 7f566ce841c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:02:56.482 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:56 smithi131 conmon[89122]: 2023-06-23T01:02:56.201+0000 7f566ce841c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:02:56.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:56.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:56 smithi131 ceph-mon[86495]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:56.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:56.629 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:56 smithi092 conmon[87350]: 2023-06-23T01:02:56.311+0000 7f190e3f41c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:02:56.630 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:56 smithi092 conmon[87350]: 2023-06-23T01:02:56.403+0000 7f190e3f41c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:02:56.889 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:56 smithi092 conmon[87350]: 2023-06-23T01:02:56.627+0000 7f190e3f41c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:02:56.890 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:56 smithi092 conmon[87350]: 2023-06-23T01:02:56.835+0000 7f190e3f41c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:02:57.086 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:56 smithi131 conmon[89122]: 2023-06-23T01:02:56.822+0000 7f566ce841c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:02:57.086 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:56 smithi131 conmon[89122]: 2023-06-23T01:02:56.993+0000 7f566ce841c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:02:57.086 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:57 smithi131 conmon[89122]: 2023-06-23T01:02:57.082+0000 7f566ce841c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:02:57.389 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:56 smithi092 conmon[87350]: 2023-06-23T01:02:56.993+0000 7f190e3f41c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:02:57.390 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:57 smithi092 conmon[87350]: 2023-06-23T01:02:57.074+0000 7f190e3f41c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:02:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.450+0000 7f7a25e4c700 1 -- 172.21.15.46:0/262559555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200fd6e0 msgr2=0x7f7a20103fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.450+0000 7f7a25e4c700 1 --2- 172.21.15.46:0/262559555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200fd6e0 0x7f7a20103fa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009de0 tx=0x7f7a18004d10 comp rx=0 tx=0).stop 2023-06-23T01:02:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.450+0000 7f7a25e4c700 1 -- 172.21.15.46:0/262559555 shutdown_connections 2023-06-23T01:02:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.450+0000 7f7a25e4c700 1 --2- 172.21.15.46:0/262559555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a200fbb40 0x7f7a200fbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.450+0000 7f7a25e4c700 1 --2- 172.21.15.46:0/262559555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200fd6e0 0x7f7a20103fa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.450+0000 7f7a25e4c700 1 --2- 172.21.15.46:0/262559555 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a200fcd50 0x7f7a200fd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.450+0000 7f7a25e4c700 1 -- 172.21.15.46:0/262559555 >> 172.21.15.46:0/262559555 conn(0x7f7a200f72c0 msgr2=0x7f7a200f96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:02:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.450+0000 7f7a25e4c700 1 -- 172.21.15.46:0/262559555 shutdown_connections 2023-06-23T01:02:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.450+0000 7f7a25e4c700 1 -- 172.21.15.46:0/262559555 wait complete. 2023-06-23T01:02:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a25e4c700 1 Processor -- start 2023-06-23T01:02:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a25e4c700 1 -- start start 2023-06-23T01:02:57.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a25e4c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a200fbb40 0x7f7a2010b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:57.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a25e4c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200fcd50 0x7f7a2010b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:57.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a25e4c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a200fd6e0 0x7f7a201128c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:57.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a25e4c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a20112ed0 con 0x7f7a200fcd50 2023-06-23T01:02:57.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a25e4c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a20113040 con 0x7f7a200fd6e0 2023-06-23T01:02:57.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a25e4c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a20113200 con 0x7f7a200fbb40 2023-06-23T01:02:57.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a1effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200fcd50 0x7f7a2010b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:57.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a1effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200fcd50 0x7f7a2010b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54510/0 (socket says 172.21.15.46:54510) 2023-06-23T01:02:57.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a1effd700 1 -- 172.21.15.46:0/1374279810 learned_addr learned my addr 172.21.15.46:0/1374279810 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:02:57.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a1ffff700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a200fd6e0 0x7f7a201128c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:57.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a1f7fe700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a200fbb40 0x7f7a2010b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:57.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.451+0000 7f7a1effd700 1 -- 172.21.15.46:0/1374279810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a200fbb40 msgr2=0x7f7a2010b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:57.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.452+0000 7f7a1effd700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a200fbb40 0x7f7a2010b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:57.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.452+0000 7f7a1effd700 1 -- 172.21.15.46:0/1374279810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a200fd6e0 msgr2=0x7f7a201128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:02:57.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.452+0000 7f7a1effd700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a200fd6e0 0x7f7a201128c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:02:57.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.452+0000 7f7a1effd700 1 -- 172.21.15.46:0/1374279810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a18009940 con 0x7f7a200fcd50 2023-06-23T01:02:57.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.452+0000 7f7a1ffff700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a200fd6e0 0x7f7a201128c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:02:57.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.452+0000 7f7a1effd700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200fcd50 0x7f7a2010b830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7a14004fd0 tx=0x7f7a14004b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:57.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.452+0000 7f7a07fff700 1 -- 172.21.15.46:0/1374279810 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14011be0 con 0x7f7a200fcd50 2023-06-23T01:02:57.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.452+0000 7f7a07fff700 1 -- 172.21.15.46:0/1374279810 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7a14013040 con 0x7f7a200fcd50 2023-06-23T01:02:57.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.452+0000 7f7a25e4c700 1 -- 172.21.15.46:0/1374279810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a201134e0 con 0x7f7a200fcd50 2023-06-23T01:02:57.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.452+0000 7f7a07fff700 1 -- 172.21.15.46:0/1374279810 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a14016390 con 0x7f7a200fcd50 2023-06-23T01:02:57.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.452+0000 7f7a25e4c700 1 -- 172.21.15.46:0/1374279810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a201139d0 con 0x7f7a200fcd50 2023-06-23T01:02:57.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.453+0000 7f7a07fff700 1 -- 172.21.15.46:0/1374279810 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f7a140167c0 con 0x7f7a200fcd50 2023-06-23T01:02:57.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.453+0000 7f7a25e4c700 1 -- 172.21.15.46:0/1374279810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a20106120 con 0x7f7a200fcd50 2023-06-23T01:02:57.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.453+0000 7f7a07fff700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a08036890 0x7f7a08038d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:02:57.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.453+0000 7f7a07fff700 1 -- 172.21.15.46:0/1374279810 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7a1404c710 con 0x7f7a200fcd50 2023-06-23T01:02:57.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.453+0000 7f7a1f7fe700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a08036890 0x7f7a08038d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:02:57.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.454+0000 7f7a1f7fe700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a08036890 0x7f7a08038d40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009b80 tx=0x7f7a1000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:02:57.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.458+0000 7f7a07fff700 1 -- 172.21.15.46:0/1374279810 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f7a1401dd10 con 0x7f7a200fcd50 2023-06-23T01:02:57.481 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:57 smithi131 conmon[89122]: 2023-06-23T01:02:57.210+0000 7f566ce841c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:02:57.482 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:57 smithi131 conmon[89122]: 2023-06-23T01:02:57.301+0000 7f566ce841c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:02:57.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:57.675+0000 7f7a25e4c700 1 -- 172.21.15.46:0/1374279810 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a2005de70 con 0x7f7a08036890 2023-06-23T01:02:57.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:57.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:57.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:57.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:57.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:57.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:57.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:57.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: Reconfiguring mgr.x (unknown last config time)... 2023-06-23T01:02:57.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-23T01:02:57.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-23T01:02:57.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:57.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:57 smithi046 ceph-mon[84949]: Reconfiguring daemon mgr.x on smithi046 2023-06-23T01:02:57.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:57.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:57.985 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:57.985 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:57.985 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:57.985 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:57.985 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:57.985 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: Reconfiguring mgr.x (unknown last config time)... 2023-06-23T01:02:57.985 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-23T01:02:57.986 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-23T01:02:57.986 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:57.986 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:57 smithi131 ceph-mon[86495]: Reconfiguring daemon mgr.x on smithi046 2023-06-23T01:02:58.139 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:57 smithi092 conmon[87350]: 2023-06-23T01:02:57.657+0000 7f190e3f41c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:02:58.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:58.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:58.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: Reconfiguring mgr.x (unknown last config time)... 2023-06-23T01:02:58.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-06-23T01:02:58.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-06-23T01:02:58.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:58.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:57 smithi092 ceph-mon[85670]: Reconfiguring daemon mgr.x on smithi046 2023-06-23T01:02:58.481 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:58 smithi131 conmon[89122]: 2023-06-23T01:02:58.183+0000 7f566ce841c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:02:58.482 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:58 smithi131 conmon[89122]: 2023-06-23T01:02:58.270+0000 7f566ce841c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:02:58.482 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:58 smithi131 conmon[89122]: 2023-06-23T01:02:58.358+0000 7f566ce841c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:02:58.639 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:58 smithi092 conmon[87350]: 2023-06-23T01:02:58.147+0000 7f190e3f41c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:02:58.640 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:58 smithi092 conmon[87350]: 2023-06-23T01:02:58.239+0000 7f190e3f41c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:02:58.640 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:58 smithi092 conmon[87350]: 2023-06-23T01:02:58.331+0000 7f190e3f41c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:02:58.763 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:58 smithi046 ceph-mon[84949]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:58.763 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:02:58.763 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:02:58.764 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:58.764 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:58 smithi046 ceph-mon[84949]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:58.764 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.765 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.765 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:58.765 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:58.765 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:58.765 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.919 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:58 smithi092 ceph-mon[85670]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:58.919 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:02:58.919 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:02:58.920 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:58.920 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:58 smithi092 ceph-mon[85670]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:58.920 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.920 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.920 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:58.920 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:58.920 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:58.921 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.981 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:58 smithi131 conmon[89122]: 2023-06-23T01:02:58.506+0000 7f566ce841c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:02:58.982 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:58 smithi131 conmon[89122]: 2023-06-23T01:02:58.605+0000 7f566ce841c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:02:58.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:58 smithi131 ceph-mon[86495]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:02:58.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:02:58.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:02:58.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:58.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:58 smithi131 ceph-mon[86495]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:02:58.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:58.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:02:58.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:02:58.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:02:58.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:02:59.180 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:58 smithi092 conmon[87350]: 2023-06-23T01:02:58.916+0000 7f190e3f41c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:02:59.180 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:59 smithi092 conmon[87350]: 2023-06-23T01:02:59.085+0000 7f190e3f41c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:02:59.273 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:59 smithi131 conmon[89122]: 2023-06-23T01:02:59.001+0000 7f566ce841c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:02:59.273 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:59 smithi131 conmon[89122]: 2023-06-23T01:02:59.090+0000 7f566ce841c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:02:59.273 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:59 smithi131 conmon[89122]: 2023-06-23T01:02:59.180+0000 7f566ce841c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:02:59.639 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:59 smithi092 conmon[87350]: 2023-06-23T01:02:59.177+0000 7f190e3f41c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:02:59.640 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:59 smithi092 conmon[87350]: 2023-06-23T01:02:59.306+0000 7f190e3f41c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:02:59.640 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:02:59 smithi092 conmon[87350]: 2023-06-23T01:02:59.397+0000 7f190e3f41c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:02:59.663 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:02:59 smithi131 conmon[89122]: 2023-06-23T01:02:59.270+0000 7f566ce841c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:02:59.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:02:59.669+0000 7f7a07fff700 1 -- 172.21.15.46:0/1374279810 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 14) v1 ==== 86210+0+0 (secure 0 0 0) 0x7f7a14020920 con 0x7f7a200fcd50 2023-06-23T01:02:59.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:59 smithi131 ceph-mon[86495]: Standby manager daemon z started 2023-06-23T01:02:59.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:59 smithi131 ceph-mon[86495]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-06-23T01:02:59.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:59 smithi131 ceph-mon[86495]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-23T01:02:59.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:59 smithi131 ceph-mon[86495]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-06-23T01:02:59.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:02:59 smithi131 ceph-mon[86495]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-23T01:03:00.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:59 smithi046 ceph-mon[84949]: Standby manager daemon z started 2023-06-23T01:03:00.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:59 smithi046 ceph-mon[84949]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-06-23T01:03:00.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:59 smithi046 ceph-mon[84949]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-23T01:03:00.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:59 smithi046 ceph-mon[84949]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-06-23T01:03:00.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:02:59 smithi046 ceph-mon[84949]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-23T01:03:00.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:59 smithi092 ceph-mon[85670]: Standby manager daemon z started 2023-06-23T01:03:00.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:59 smithi092 ceph-mon[85670]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-06-23T01:03:00.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:59 smithi092 ceph-mon[85670]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-23T01:03:00.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:59 smithi092 ceph-mon[85670]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-06-23T01:03:00.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:02:59 smithi092 ceph-mon[85670]: from='mgr.? 172.21.15.131:0/1122725614' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-23T01:03:00.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:03:00 smithi092 conmon[87350]: 2023-06-23T01:03:00.274+0000 7f190e3f41c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:03:00.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:03:00 smithi092 conmon[87350]: 2023-06-23T01:03:00.362+0000 7f190e3f41c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:03:00.601 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:03:00 smithi092 conmon[87350]: 2023-06-23T01:03:00.450+0000 7f190e3f41c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:03:00.889 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:03:00 smithi092 conmon[87350]: 2023-06-23T01:03:00.598+0000 7f190e3f41c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:03:00.890 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:03:00 smithi092 conmon[87350]: 2023-06-23T01:03:00.696+0000 7f190e3f41c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:03:00.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:00 smithi092 ceph-mon[85670]: mgrmap e14: x(active, since 63s), standbys: z 2023-06-23T01:03:00.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:00 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-06-23T01:03:00.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:00 smithi092 ceph-mon[85670]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:00.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:00 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2124602582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f28767ce-a780-4e50-b118-37a8c275c0e9"}]: dispatch 2023-06-23T01:03:00.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:00 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2124602582' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f28767ce-a780-4e50-b118-37a8c275c0e9"}]': finished 2023-06-23T01:03:00.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:00 smithi092 ceph-mon[85670]: osdmap e5: 1 total, 0 up, 1 in 2023-06-23T01:03:00.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:00 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:00.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:00 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/3313027220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:03:00.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:00 smithi131 ceph-mon[86495]: mgrmap e14: x(active, since 63s), standbys: z 2023-06-23T01:03:00.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:00 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-06-23T01:03:00.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:00 smithi131 ceph-mon[86495]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:00.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:00 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2124602582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f28767ce-a780-4e50-b118-37a8c275c0e9"}]: dispatch 2023-06-23T01:03:00.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:00 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2124602582' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f28767ce-a780-4e50-b118-37a8c275c0e9"}]': finished 2023-06-23T01:03:00.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:00 smithi131 ceph-mon[86495]: osdmap e5: 1 total, 0 up, 1 in 2023-06-23T01:03:00.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:00 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:00.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:00 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/3313027220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:03:01.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:00 smithi046 ceph-mon[84949]: mgrmap e14: x(active, since 63s), standbys: z 2023-06-23T01:03:01.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:00 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-06-23T01:03:01.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:00 smithi046 ceph-mon[84949]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:01.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:00 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2124602582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f28767ce-a780-4e50-b118-37a8c275c0e9"}]: dispatch 2023-06-23T01:03:01.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:00 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2124602582' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f28767ce-a780-4e50-b118-37a8c275c0e9"}]': finished 2023-06-23T01:03:01.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:00 smithi046 ceph-mon[84949]: osdmap e5: 1 total, 0 up, 1 in 2023-06-23T01:03:01.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:00 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:01.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:00 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3313027220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:03:01.354 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:03:01 smithi092 conmon[87350]: 2023-06-23T01:03:01.084+0000 7f190e3f41c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:03:01.355 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:03:01 smithi092 conmon[87350]: 2023-06-23T01:03:01.173+0000 7f190e3f41c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:03:01.355 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:03:01 smithi092 conmon[87350]: 2023-06-23T01:03:01.262+0000 7f190e3f41c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:03:01.639 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:03:01 smithi092 conmon[87350]: 2023-06-23T01:03:01.352+0000 7f190e3f41c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:03:01.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:01.678+0000 7f7a07fff700 1 -- 172.21.15.46:0/1374279810 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 15) v1 ==== 129145+0+0 (secure 0 0 0) 0x7f7a140173f0 con 0x7f7a200fcd50 2023-06-23T01:03:01.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:01 smithi131 ceph-mon[86495]: Standby manager daemon y started 2023-06-23T01:03:01.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:01 smithi131 ceph-mon[86495]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-06-23T01:03:01.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:01 smithi131 ceph-mon[86495]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-23T01:03:01.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:01 smithi131 ceph-mon[86495]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-06-23T01:03:01.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:01 smithi131 ceph-mon[86495]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-23T01:03:02.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:01 smithi046 ceph-mon[84949]: Standby manager daemon y started 2023-06-23T01:03:02.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:01 smithi046 ceph-mon[84949]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-06-23T01:03:02.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:01 smithi046 ceph-mon[84949]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-23T01:03:02.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:01 smithi046 ceph-mon[84949]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-06-23T01:03:02.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:01 smithi046 ceph-mon[84949]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-23T01:03:02.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:01 smithi092 ceph-mon[85670]: Standby manager daemon y started 2023-06-23T01:03:02.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:01 smithi092 ceph-mon[85670]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-06-23T01:03:02.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:01 smithi092 ceph-mon[85670]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-06-23T01:03:02.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:01 smithi092 ceph-mon[85670]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-06-23T01:03:02.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:01 smithi092 ceph-mon[85670]: from='mgr.? 172.21.15.92:0/1660885835' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-06-23T01:03:02.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:02 smithi131 ceph-mon[86495]: mgrmap e15: x(active, since 65s), standbys: z, y 2023-06-23T01:03:02.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-06-23T01:03:02.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:02 smithi131 ceph-mon[86495]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:03.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:02 smithi046 ceph-mon[84949]: mgrmap e15: x(active, since 65s), standbys: z, y 2023-06-23T01:03:03.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-06-23T01:03:03.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:02 smithi046 ceph-mon[84949]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:03.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:02 smithi092 ceph-mon[85670]: mgrmap e15: x(active, since 65s), standbys: z, y 2023-06-23T01:03:03.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-06-23T01:03:03.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:02 smithi092 ceph-mon[85670]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:05.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:04 smithi046 ceph-mon[84949]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:05.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:04 smithi092 ceph-mon[85670]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:05.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:04 smithi131 ceph-mon[86495]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:07.124 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:06 smithi046 ceph-mon[84949]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:07.124 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-06-23T01:03:07.124 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:07.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:06 smithi092 ceph-mon[85670]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:07.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:06 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-06-23T01:03:07.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:06 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:07.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:06 smithi131 ceph-mon[86495]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:07.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:06 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-06-23T01:03:07.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:06 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:08.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:07 smithi092 ceph-mon[85670]: Deploying daemon osd.0 on smithi046 2023-06-23T01:03:08.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:07 smithi131 ceph-mon[86495]: Deploying daemon osd.0 on smithi046 2023-06-23T01:03:08.255 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:07 smithi046 ceph-mon[84949]: Deploying daemon osd.0 on smithi046 2023-06-23T01:03:09.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:08 smithi092 ceph-mon[85670]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:09.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:08 smithi131 ceph-mon[86495]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:09.297 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:08 smithi046 ceph-mon[84949]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:11.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:10 smithi046 ceph-mon[84949]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:11.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:10 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:11.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:10 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:11.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:10 smithi092 ceph-mon[85670]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:11.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:10 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:11.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:10 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:11.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:10 smithi131 ceph-mon[86495]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:11.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:10 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:11.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:10 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:11.995 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:11 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:12.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:11 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:12.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:11 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:13.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:12 smithi092 ceph-mon[85670]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:13.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:12 smithi046 ceph-mon[84949]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:13.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:12 smithi131 ceph-mon[86495]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:13.375 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 0 on host 'smithi046' 2023-06-23T01:03:13.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.372+0000 7f7a07fff700 1 -- 172.21.15.46:0/1374279810 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7a2005de70 con 0x7f7a08036890 2023-06-23T01:03:13.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 -- 172.21.15.46:0/1374279810 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a08036890 msgr2=0x7f7a08038d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:13.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a08036890 0x7f7a08038d40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009b80 tx=0x7f7a1000b040 comp rx=0 tx=0).stop 2023-06-23T01:03:13.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 -- 172.21.15.46:0/1374279810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200fcd50 msgr2=0x7f7a2010b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:13.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200fcd50 0x7f7a2010b830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7a14004fd0 tx=0x7f7a14004b80 comp rx=0 tx=0).stop 2023-06-23T01:03:13.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 -- 172.21.15.46:0/1374279810 shutdown_connections 2023-06-23T01:03:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a08036890 0x7f7a08038d40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a200fd6e0 0x7f7a201128c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:13.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a200fbb40 0x7f7a2010b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 --2- 172.21.15.46:0/1374279810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200fcd50 0x7f7a2010b830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 -- 172.21.15.46:0/1374279810 >> 172.21.15.46:0/1374279810 conn(0x7f7a200f72c0 msgr2=0x7f7a200f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:03:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 -- 172.21.15.46:0/1374279810 shutdown_connections 2023-06-23T01:03:13.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:03:13.374+0000 7f7a25e4c700 1 -- 172.21.15.46:0/1374279810 wait complete. 2023-06-23T01:03:13.791 DEBUG:teuthology.orchestra.run.smithi046:osd.0> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.0.service 2023-06-23T01:03:13.793 INFO:tasks.cephadm:Deploying osd.1 on smithi092 with /dev/vg_nvme/lv_4... 2023-06-23T01:03:13.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-06-23T01:03:13.820 INFO:journalctl@ceph.osd.0.smithi046.stdout:-- Logs begin at Fri 2023-06-23 00:44:15 UTC. -- 2023-06-23T01:03:14.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:13 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:13 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:13 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:13 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:13 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:13 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:13 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:13 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:13 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:13 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:13 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:13 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:14.390 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-23T01:03:15.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:14 smithi046 ceph-mon[84949]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:15.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:14 smithi046 ceph-mon[84949]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-06-23T01:03:15.069 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:03:14 smithi046 conmon[101888]: 2023-06-23T01:03:14.758+0000 7fa5d35a3540 -1 osd.0 0 log_to_monitors true 2023-06-23T01:03:15.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:14 smithi131 ceph-mon[86495]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:15.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:14 smithi131 ceph-mon[86495]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-06-23T01:03:15.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:14 smithi092 ceph-mon[85670]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:15.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:14 smithi092 ceph-mon[85670]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-06-23T01:03:15.922 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-06-23T01:03:15.922 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-06-23T01:03:15.922 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2023-06-23T01:03:15.922 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2023-06-23T01:03:15.922 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114561 s, 915 MB/s 2023-06-23T01:03:15.922 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2023-06-23T01:03:16.337 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi092:vg_nvme/lv_4 2023-06-23T01:03:16.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-06-23T01:03:16.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: osdmap e6: 1 total, 0 up, 1 in 2023-06-23T01:03:16.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:16.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:03:16.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:16.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:03:16.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:16.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:16.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:16.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-06-23T01:03:16.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: osdmap e6: 1 total, 0 up, 1 in 2023-06-23T01:03:16.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:16.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:03:16.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:16.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:03:16.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:16.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:16.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.734 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:16.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-06-23T01:03:16.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: osdmap e6: 1 total, 0 up, 1 in 2023-06-23T01:03:16.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:16.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:03:16.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:16.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:03:16.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:16.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:16.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:16.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:16.820 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:03:16 smithi046 conmon[101888]: 2023-06-23T01:03:16.351+0000 7fa5c9d88700 -1 osd.0 0 waiting for initial osdmap 2023-06-23T01:03:16.820 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:03:16 smithi046 conmon[101888]: 2023-06-23T01:03:16.355+0000 7fa5c4382700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:03:16.952 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-23T01:03:17.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:17 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi046 2023-06-23T01:03:17.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:17 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi046 to 17173M 2023-06-23T01:03:17.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:17 smithi092 ceph-mon[85670]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:03:17.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:17 smithi092 ceph-mon[85670]: osdmap e7: 1 total, 0 up, 1 in 2023-06-23T01:03:17.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:17 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:17.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:17 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:17.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:17 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:17.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:17 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:17.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:17 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:17.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:17 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:17.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:17 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi046 2023-06-23T01:03:17.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:17 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi046 to 17173M 2023-06-23T01:03:17.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:17 smithi131 ceph-mon[86495]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:03:17.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:17 smithi131 ceph-mon[86495]: osdmap e7: 1 total, 0 up, 1 in 2023-06-23T01:03:17.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:17 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:17.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:17 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:17.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:17 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:17.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:17 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:17.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:17 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:17.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:17 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:17.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:17 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi046 2023-06-23T01:03:17.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:17 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi046 to 17173M 2023-06-23T01:03:17.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:17 smithi046 ceph-mon[84949]: from='osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:03:17.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:17 smithi046 ceph-mon[84949]: osdmap e7: 1 total, 0 up, 1 in 2023-06-23T01:03:17.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:17 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:17.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:17 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:17.821 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:17 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:17.821 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:17 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:17.821 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:17 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:17.821 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:17 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:18.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.161+0000 7ff28d8af700 1 -- 172.21.15.92:0/598265506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2880fbb40 msgr2=0x7ff2880fbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:18.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.161+0000 7ff28d8af700 1 --2- 172.21.15.92:0/598265506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2880fbb40 0x7ff2880fbf50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff2780049c0 tx=0x7ff278009e30 comp rx=0 tx=0).stop 2023-06-23T01:03:18.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.161+0000 7ff28d8af700 1 -- 172.21.15.92:0/598265506 shutdown_connections 2023-06-23T01:03:18.167 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.161+0000 7ff28d8af700 1 --2- 172.21.15.92:0/598265506 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2880fbb40 0x7ff2880fbf50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:18.167 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.161+0000 7ff28d8af700 1 --2- 172.21.15.92:0/598265506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2880fd6e0 0x7ff288103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:18.167 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.161+0000 7ff28d8af700 1 --2- 172.21.15.92:0/598265506 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2880fcd50 0x7ff2880fd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:18.167 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.161+0000 7ff28d8af700 1 -- 172.21.15.92:0/598265506 >> 172.21.15.92:0/598265506 conn(0x7ff2880f72c0 msgr2=0x7ff2880f96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:03:18.167 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.161+0000 7ff28d8af700 1 -- 172.21.15.92:0/598265506 shutdown_connections 2023-06-23T01:03:18.167 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.161+0000 7ff28d8af700 1 -- 172.21.15.92:0/598265506 wait complete. 2023-06-23T01:03:18.167 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.161+0000 7ff28d8af700 1 Processor -- start 2023-06-23T01:03:18.168 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.161+0000 7ff28d8af700 1 -- start start 2023-06-23T01:03:18.168 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff28d8af700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2880fbb40 0x7ff28810d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:03:18.168 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff28d8af700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2880fcd50 0x7ff28810ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:03:18.168 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff28d8af700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2880fd6e0 0x7ff288114e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:03:18.168 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff28d8af700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288115490 con 0x7ff2880fd6e0 2023-06-23T01:03:18.168 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff28d8af700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff288115600 con 0x7ff2880fcd50 2023-06-23T01:03:18.168 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff28d8af700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2881157c0 con 0x7ff2880fbb40 2023-06-23T01:03:18.169 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2880fcd50 0x7ff28810ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:03:18.169 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2880fcd50 0x7ff28810ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:42700/0 (socket says 172.21.15.92:42700) 2023-06-23T01:03:18.169 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27e7fc700 1 -- 172.21.15.92:0/3428318910 learned_addr learned my addr 172.21.15.92:0/3428318910 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:03:18.169 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff2877fe700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2880fd6e0 0x7ff288114e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:03:18.169 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff286ffd700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2880fbb40 0x7ff28810d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:03:18.169 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27e7fc700 1 -- 172.21.15.92:0/3428318910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2880fbb40 msgr2=0x7ff28810d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:18.169 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27e7fc700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2880fbb40 0x7ff28810d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:18.169 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27e7fc700 1 -- 172.21.15.92:0/3428318910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2880fd6e0 msgr2=0x7ff288114e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:18.170 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27e7fc700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2880fd6e0 0x7ff288114e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:18.170 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27e7fc700 1 -- 172.21.15.92:0/3428318910 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff278005040 con 0x7ff2880fcd50 2023-06-23T01:03:18.170 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27e7fc700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2880fcd50 0x7ff28810ddf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff27000b810 tx=0x7ff270009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:03:18.170 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27ffff700 1 -- 172.21.15.92:0/3428318910 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27000f510 con 0x7ff2880fcd50 2023-06-23T01:03:18.170 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff28d8af700 1 -- 172.21.15.92:0/3428318910 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff288115aa0 con 0x7ff2880fcd50 2023-06-23T01:03:18.170 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27ffff700 1 -- 172.21.15.92:0/3428318910 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff27000fb10 con 0x7ff2880fcd50 2023-06-23T01:03:18.170 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff27ffff700 1 -- 172.21.15.92:0/3428318910 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff270011b80 con 0x7ff2880fcd50 2023-06-23T01:03:18.170 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.162+0000 7ff28d8af700 1 -- 172.21.15.92:0/3428318910 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff288115f90 con 0x7ff2880fcd50 2023-06-23T01:03:18.171 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.163+0000 7ff28d8af700 1 -- 172.21.15.92:0/3428318910 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff288107210 con 0x7ff2880fcd50 2023-06-23T01:03:18.171 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.165+0000 7ff27ffff700 1 -- 172.21.15.92:0/3428318910 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 15) v1 ==== 129145+0+0 (secure 0 0 0) 0x7ff27000fc80 con 0x7ff2880fcd50 2023-06-23T01:03:18.172 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.166+0000 7ff27ffff700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff27409b5f0 0x7ff27409daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:03:18.172 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.166+0000 7ff27ffff700 1 -- 172.21.15.92:0/3428318910 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1763+0+0 (secure 0 0 0) 0x7ff2700c3a80 con 0x7ff2880fcd50 2023-06-23T01:03:18.172 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.166+0000 7ff286ffd700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff27409b5f0 0x7ff27409daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:03:18.172 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.166+0000 7ff286ffd700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff27409b5f0 0x7ff27409daa0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff278009b30 tx=0x7ff278009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:03:18.172 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.169+0000 7ff27ffff700 1 -- 172.21.15.92:0/3428318910 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7ff270090de0 con 0x7ff2880fcd50 2023-06-23T01:03:18.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:18.360+0000 7ff28d8af700 1 -- 172.21.15.92:0/3428318910 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff288065b70 con 0x7ff27409b5f0 2023-06-23T01:03:18.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:18 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:03:18.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:18 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:03:18.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:18 smithi092 ceph-mon[85670]: osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010] boot 2023-06-23T01:03:18.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:18 smithi092 ceph-mon[85670]: osdmap e8: 1 total, 1 up, 1 in 2023-06-23T01:03:18.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:18.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:18 smithi092 ceph-mon[85670]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:18.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:18 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:03:18.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:18 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:03:18.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:18 smithi131 ceph-mon[86495]: osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010] boot 2023-06-23T01:03:18.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:18 smithi131 ceph-mon[86495]: osdmap e8: 1 total, 1 up, 1 in 2023-06-23T01:03:18.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:18.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:18 smithi131 ceph-mon[86495]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:18.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:18 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:03:18.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:18 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:03:18.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:18 smithi046 ceph-mon[84949]: osd.0 [v2:172.21.15.46:6802/709285010,v1:172.21.15.46:6803/709285010] boot 2023-06-23T01:03:18.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:18 smithi046 ceph-mon[84949]: osdmap e8: 1 total, 1 up, 1 in 2023-06-23T01:03:18.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-06-23T01:03:18.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:18 smithi046 ceph-mon[84949]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-06-23T01:03:19.583 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:19 smithi092 ceph-mon[85670]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:03:19.584 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:19 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:03:19.584 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:19 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:03:19.584 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:19 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:19.584 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:19 smithi092 ceph-mon[85670]: osdmap e9: 1 total, 1 up, 1 in 2023-06-23T01:03:19.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:19 smithi131 ceph-mon[86495]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:03:19.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:19 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:03:19.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:19 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:03:19.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:19 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:19.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:19 smithi131 ceph-mon[86495]: osdmap e9: 1 total, 1 up, 1 in 2023-06-23T01:03:19.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:19 smithi046 ceph-mon[84949]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:03:19.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:03:19.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:03:19.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:19.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:19 smithi046 ceph-mon[84949]: osdmap e9: 1 total, 1 up, 1 in 2023-06-23T01:03:20.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:20 smithi092 ceph-mon[85670]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:20.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:20 smithi131 ceph-mon[86495]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:20.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:20 smithi046 ceph-mon[84949]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:21.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:21 smithi092 ceph-mon[85670]: from='client.? 172.21.15.92:0/541051884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d4c4c36-d4a1-4b96-9059-adffee04401a"}]: dispatch 2023-06-23T01:03:21.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:21 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d4c4c36-d4a1-4b96-9059-adffee04401a"}]: dispatch 2023-06-23T01:03:21.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:21 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d4c4c36-d4a1-4b96-9059-adffee04401a"}]': finished 2023-06-23T01:03:21.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:21 smithi092 ceph-mon[85670]: osdmap e10: 2 total, 1 up, 2 in 2023-06-23T01:03:21.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:21.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:21 smithi092 ceph-mon[85670]: from='client.? 172.21.15.92:0/699571638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:03:21.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:21 smithi131 ceph-mon[86495]: from='client.? 172.21.15.92:0/541051884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d4c4c36-d4a1-4b96-9059-adffee04401a"}]: dispatch 2023-06-23T01:03:21.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:21 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d4c4c36-d4a1-4b96-9059-adffee04401a"}]: dispatch 2023-06-23T01:03:21.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:21 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d4c4c36-d4a1-4b96-9059-adffee04401a"}]': finished 2023-06-23T01:03:21.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:21 smithi131 ceph-mon[86495]: osdmap e10: 2 total, 1 up, 2 in 2023-06-23T01:03:21.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:21.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:21 smithi131 ceph-mon[86495]: from='client.? 172.21.15.92:0/699571638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:03:21.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:21 smithi046 ceph-mon[84949]: from='client.? 172.21.15.92:0/541051884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d4c4c36-d4a1-4b96-9059-adffee04401a"}]: dispatch 2023-06-23T01:03:21.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:21 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d4c4c36-d4a1-4b96-9059-adffee04401a"}]: dispatch 2023-06-23T01:03:21.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:21 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d4c4c36-d4a1-4b96-9059-adffee04401a"}]': finished 2023-06-23T01:03:21.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:21 smithi046 ceph-mon[84949]: osdmap e10: 2 total, 1 up, 2 in 2023-06-23T01:03:21.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:21.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:21 smithi046 ceph-mon[84949]: from='client.? 172.21.15.92:0/699571638' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:03:22.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:22 smithi092 ceph-mon[85670]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:22.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:22 smithi131 ceph-mon[86495]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:22.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:22 smithi046 ceph-mon[84949]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:25.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:24 smithi092 ceph-mon[85670]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:25.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:24 smithi131 ceph-mon[86495]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:25.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:24 smithi046 ceph-mon[84949]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:27.135 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:26 smithi092 ceph-mon[85670]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:27.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:26 smithi131 ceph-mon[86495]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:27.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:26 smithi046 ceph-mon[84949]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:28.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-06-23T01:03:28.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:28.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-06-23T01:03:28.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:28.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-06-23T01:03:28.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:28.996 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:28 smithi092 ceph-mon[85670]: Deploying daemon osd.1 on smithi092 2023-06-23T01:03:28.996 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:28 smithi092 ceph-mon[85670]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:29.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:28 smithi131 ceph-mon[86495]: Deploying daemon osd.1 on smithi092 2023-06-23T01:03:29.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:28 smithi131 ceph-mon[86495]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:29.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:28 smithi046 ceph-mon[84949]: Deploying daemon osd.1 on smithi092 2023-06-23T01:03:29.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:28 smithi046 ceph-mon[84949]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:31.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:30 smithi092 ceph-mon[85670]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:31.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:30 smithi131 ceph-mon[86495]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:31.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:30 smithi046 ceph-mon[84949]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:32.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:32 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:32.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:32 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:32.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:32 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:32.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:32 smithi092 ceph-mon[85670]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:32.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:32 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:32.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:32 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:32.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:32 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:32.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:32 smithi131 ceph-mon[86495]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:33.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:32 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:33.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:32 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:33.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:32 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:33.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:32 smithi046 ceph-mon[84949]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:34.478 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.476+0000 7ff27ffff700 1 -- 172.21.15.92:0/3428318910 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff288065b70 con 0x7ff27409b5f0 2023-06-23T01:03:34.479 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 1 on host 'smithi092' 2023-06-23T01:03:34.480 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 -- 172.21.15.92:0/3428318910 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff27409b5f0 msgr2=0x7ff27409daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:34.480 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff27409b5f0 0x7ff27409daa0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff278009b30 tx=0x7ff278009e30 comp rx=0 tx=0).stop 2023-06-23T01:03:34.481 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 -- 172.21.15.92:0/3428318910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2880fcd50 msgr2=0x7ff28810ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:34.481 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2880fcd50 0x7ff28810ddf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff27000b810 tx=0x7ff270009250 comp rx=0 tx=0).stop 2023-06-23T01:03:34.481 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 -- 172.21.15.92:0/3428318910 shutdown_connections 2023-06-23T01:03:34.481 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2880fd6e0 0x7ff288114e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:34.481 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff2880fbb40 0x7ff28810d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:34.481 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff27409b5f0 0x7ff27409daa0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:34.482 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 --2- 172.21.15.92:0/3428318910 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2880fcd50 0x7ff28810ddf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:34.482 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 -- 172.21.15.92:0/3428318910 >> 172.21.15.92:0/3428318910 conn(0x7ff2880f72c0 msgr2=0x7ff288065460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:03:34.482 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 -- 172.21.15.92:0/3428318910 shutdown_connections 2023-06-23T01:03:34.482 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:03:34.478+0000 7ff28d8af700 1 -- 172.21.15.92:0/3428318910 wait complete. 2023-06-23T01:03:34.714 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.714 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.714 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:34.714 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:34.714 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:34 smithi092 ceph-mon[85670]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:34.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.715 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:34.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:34.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:34.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:34 smithi046 ceph-mon[84949]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:34.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:34.900 DEBUG:teuthology.orchestra.run.smithi092:osd.1> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.1.service 2023-06-23T01:03:34.903 INFO:tasks.cephadm:Deploying osd.2 on smithi131 with /dev/vg_nvme/lv_4... 2023-06-23T01:03:34.903 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-06-23T01:03:34.930 INFO:journalctl@ceph.osd.1.smithi092.stdout:-- Logs begin at Fri 2023-06-23 00:44:08 UTC. -- 2023-06-23T01:03:34.931 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.931 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.931 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:34.931 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:34.932 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.932 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:34 smithi131 ceph-mon[86495]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:34.932 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.933 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:34.933 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:35.488 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:03:35.780 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:35 smithi092 ceph-mon[85670]: from='osd.1 [v2:172.21.15.92:6800/4266072183,v1:172.21.15.92:6801/4266072183]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-23T01:03:35.780 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:35 smithi092 ceph-mon[85670]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-23T01:03:35.780 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:03:35 smithi092 conmon[92152]: 2023-06-23T01:03:35.505+0000 7f5bdf2a0540 -1 osd.1 0 log_to_monitors true 2023-06-23T01:03:35.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:35 smithi046 ceph-mon[84949]: from='osd.1 [v2:172.21.15.92:6800/4266072183,v1:172.21.15.92:6801/4266072183]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-23T01:03:35.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:35 smithi046 ceph-mon[84949]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-23T01:03:35.819 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:35 smithi131 ceph-mon[86495]: from='osd.1 [v2:172.21.15.92:6800/4266072183,v1:172.21.15.92:6801/4266072183]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-23T01:03:35.820 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:35 smithi131 ceph-mon[86495]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-06-23T01:03:36.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:36 smithi046 ceph-mon[84949]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-06-23T01:03:36.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:36 smithi046 ceph-mon[84949]: osdmap e11: 2 total, 1 up, 2 in 2023-06-23T01:03:36.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:36.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:36 smithi046 ceph-mon[84949]: from='osd.1 [v2:172.21.15.92:6800/4266072183,v1:172.21.15.92:6801/4266072183]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:03:36.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:36 smithi046 ceph-mon[84949]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:03:36.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:36 smithi046 ceph-mon[84949]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:36.846 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:03:36 smithi092 conmon[92152]: 2023-06-23T01:03:36.565+0000 7f5bd5a85700 -1 osd.1 0 waiting for initial osdmap 2023-06-23T01:03:36.847 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:03:36 smithi092 conmon[92152]: 2023-06-23T01:03:36.568+0000 7f5bcf87e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:03:36.847 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:36 smithi092 ceph-mon[85670]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-06-23T01:03:36.847 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:36 smithi092 ceph-mon[85670]: osdmap e11: 2 total, 1 up, 2 in 2023-06-23T01:03:36.847 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:36.848 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:36 smithi092 ceph-mon[85670]: from='osd.1 [v2:172.21.15.92:6800/4266072183,v1:172.21.15.92:6801/4266072183]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:03:36.848 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:36 smithi092 ceph-mon[85670]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:03:36.848 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:36 smithi092 ceph-mon[85670]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:36.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:36 smithi131 ceph-mon[86495]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-06-23T01:03:36.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:36 smithi131 ceph-mon[86495]: osdmap e11: 2 total, 1 up, 2 in 2023-06-23T01:03:36.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:36.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:36 smithi131 ceph-mon[86495]: from='osd.1 [v2:172.21.15.92:6800/4266072183,v1:172.21.15.92:6801/4266072183]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:03:36.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:36 smithi131 ceph-mon[86495]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:03:36.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:36 smithi131 ceph-mon[86495]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:37.066 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-06-23T01:03:37.067 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-06-23T01:03:37.067 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2023-06-23T01:03:37.067 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2023-06-23T01:03:37.067 INFO:teuthology.orchestra.run.smithi131.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013129 s, 799 MB/s 2023-06-23T01:03:37.067 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2023-06-23T01:03:37.432 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi131:vg_nvme/lv_4 2023-06-23T01:03:37.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:37 smithi131 ceph-mon[86495]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:03:37.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:37 smithi131 ceph-mon[86495]: osdmap e12: 2 total, 1 up, 2 in 2023-06-23T01:03:37.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:37 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:37.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:37 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:37.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:37 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:37.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:37 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:37.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:37 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:03:37.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:37 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:37.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:37 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:37.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:37 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:37.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:37 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:37.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:37 smithi092 ceph-mon[85670]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:03:37.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:37 smithi092 ceph-mon[85670]: osdmap e12: 2 total, 1 up, 2 in 2023-06-23T01:03:37.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:37 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:37.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:37 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:37.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:37 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:37.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:37 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:37.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:37 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:03:37.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:37 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:37.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:37 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:37.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:37 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:37.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:37 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:38.051 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:03:38.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:37 smithi046 ceph-mon[84949]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:03:38.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:37 smithi046 ceph-mon[84949]: osdmap e12: 2 total, 1 up, 2 in 2023-06-23T01:03:38.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:37 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:38.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:37 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:38.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:37 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:38.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:37 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:38.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:37 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:03:38.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:37 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:38.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:37 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:38.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:37 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:38.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:37 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:38.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:38 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:03:38.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:38 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:03:38.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:38 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi092 2023-06-23T01:03:38.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:38 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi092 to 17168M 2023-06-23T01:03:38.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:38 smithi092 ceph-mon[85670]: osd.1 [v2:172.21.15.92:6800/4266072183,v1:172.21.15.92:6801/4266072183] boot 2023-06-23T01:03:38.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:38 smithi092 ceph-mon[85670]: osdmap e13: 2 total, 2 up, 2 in 2023-06-23T01:03:38.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:38.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:38 smithi092 ceph-mon[85670]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:38.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:38 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:03:38.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:38 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:03:38.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:38 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi092 2023-06-23T01:03:38.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:38 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi092 to 17168M 2023-06-23T01:03:38.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:38 smithi131 ceph-mon[86495]: osd.1 [v2:172.21.15.92:6800/4266072183,v1:172.21.15.92:6801/4266072183] boot 2023-06-23T01:03:38.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:38 smithi131 ceph-mon[86495]: osdmap e13: 2 total, 2 up, 2 in 2023-06-23T01:03:38.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:38 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:38.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:38 smithi131 ceph-mon[86495]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:39.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:38 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:03:39.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:38 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:03:39.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:38 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi092 2023-06-23T01:03:39.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:38 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi092 to 17168M 2023-06-23T01:03:39.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:38 smithi046 ceph-mon[84949]: osd.1 [v2:172.21.15.92:6800/4266072183,v1:172.21.15.92:6801/4266072183] boot 2023-06-23T01:03:39.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:38 smithi046 ceph-mon[84949]: osdmap e13: 2 total, 2 up, 2 in 2023-06-23T01:03:39.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:38 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-06-23T01:03:39.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:38 smithi046 ceph-mon[84949]: pgmap v54: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-06-23T01:03:39.248 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.242+0000 7f4c99728700 1 -- 172.21.15.131:0/3481245198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4c940fd7a0 msgr2=0x7f4c94104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:39.249 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.242+0000 7f4c99728700 1 --2- 172.21.15.131:0/3481245198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4c940fd7a0 0x7f4c94104040 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009d60 tx=0x7f4c8c004d10 comp rx=0 tx=0).stop 2023-06-23T01:03:39.249 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.242+0000 7f4c99728700 1 -- 172.21.15.131:0/3481245198 shutdown_connections 2023-06-23T01:03:39.249 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.242+0000 7f4c99728700 1 --2- 172.21.15.131:0/3481245198 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c940fbb40 0x7f4c940fbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:39.250 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.242+0000 7f4c99728700 1 --2- 172.21.15.131:0/3481245198 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4c940fd7a0 0x7f4c94104040 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:39.250 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.242+0000 7f4c99728700 1 --2- 172.21.15.131:0/3481245198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c940fcdf0 0x7f4c940fd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:39.250 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.242+0000 7f4c99728700 1 -- 172.21.15.131:0/3481245198 >> 172.21.15.131:0/3481245198 conn(0x7f4c940f72a0 msgr2=0x7f4c940f96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:03:39.251 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.242+0000 7f4c99728700 1 -- 172.21.15.131:0/3481245198 shutdown_connections 2023-06-23T01:03:39.251 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.242+0000 7f4c99728700 1 -- 172.21.15.131:0/3481245198 wait complete. 2023-06-23T01:03:39.252 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.243+0000 7f4c99728700 1 Processor -- start 2023-06-23T01:03:39.253 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.243+0000 7f4c99728700 1 -- start start 2023-06-23T01:03:39.253 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.243+0000 7f4c99728700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c940fbb40 0x7f4c9410d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:03:39.254 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.243+0000 7f4c99728700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c940fcdf0 0x7f4c9410dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:03:39.254 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.243+0000 7f4c99728700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4c940fd7a0 0x7f4c94114d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:03:39.254 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.243+0000 7f4c99728700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c94115390 con 0x7f4c940fbb40 2023-06-23T01:03:39.254 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.243+0000 7f4c99728700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c94115500 con 0x7f4c940fcdf0 2023-06-23T01:03:39.255 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.243+0000 7f4c99728700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c941156c0 con 0x7f4c940fd7a0 2023-06-23T01:03:39.255 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.243+0000 7f4c937fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4c940fd7a0 0x7f4c94114d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:03:39.255 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c937fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4c940fd7a0 0x7f4c94114d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48544/0 (socket says 172.21.15.131:48544) 2023-06-23T01:03:39.256 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c937fe700 1 -- 172.21.15.131:0/446432386 learned_addr learned my addr 172.21.15.131:0/446432386 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-06-23T01:03:39.256 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c92ffd700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c940fbb40 0x7f4c9410d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:03:39.256 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c927fc700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c940fcdf0 0x7f4c9410dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:03:39.256 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c937fe700 1 -- 172.21.15.131:0/446432386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c940fcdf0 msgr2=0x7f4c9410dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:39.257 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c937fe700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c940fcdf0 0x7f4c9410dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:39.257 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c937fe700 1 -- 172.21.15.131:0/446432386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c940fbb40 msgr2=0x7f4c9410d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:39.257 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c937fe700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c940fbb40 0x7f4c9410d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:39.258 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c937fe700 1 -- 172.21.15.131:0/446432386 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c009940 con 0x7f4c940fd7a0 2023-06-23T01:03:39.258 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c937fe700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4c940fd7a0 0x7f4c94114d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009fd0 tx=0x7f4c8c004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:03:39.258 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c7b7fe700 1 -- 172.21.15.131:0/446432386 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c00e030 con 0x7f4c940fd7a0 2023-06-23T01:03:39.258 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c7b7fe700 1 -- 172.21.15.131:0/446432386 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4c8c01caf0 con 0x7f4c940fd7a0 2023-06-23T01:03:39.259 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c7b7fe700 1 -- 172.21.15.131:0/446432386 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c012870 con 0x7f4c940fd7a0 2023-06-23T01:03:39.259 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c99728700 1 -- 172.21.15.131:0/446432386 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c94115940 con 0x7f4c940fd7a0 2023-06-23T01:03:39.259 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.244+0000 7f4c99728700 1 -- 172.21.15.131:0/446432386 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c94115da0 con 0x7f4c940fd7a0 2023-06-23T01:03:39.260 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.245+0000 7f4c99728700 1 -- 172.21.15.131:0/446432386 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c94107350 con 0x7f4c940fd7a0 2023-06-23T01:03:39.261 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.246+0000 7f4c7b7fe700 1 -- 172.21.15.131:0/446432386 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 15) v1 ==== 129145+0+0 (secure 0 0 0) 0x7f4c8c012ca0 con 0x7f4c940fd7a0 2023-06-23T01:03:39.261 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.247+0000 7f4c7b7fe700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4c7c09b6f0 0x7f4c7c09dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:03:39.262 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.247+0000 7f4c7b7fe700 1 -- 172.21.15.131:0/446432386 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2305+0+0 (secure 0 0 0) 0x7f4c8c0c5a90 con 0x7f4c940fd7a0 2023-06-23T01:03:39.262 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.247+0000 7f4c92ffd700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4c7c09b6f0 0x7f4c7c09dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:03:39.262 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.247+0000 7f4c92ffd700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4c7c09b6f0 0x7f4c7c09dba0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009bf0 tx=0x7f4c840091a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:03:39.262 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.251+0000 7f4c7b7fe700 1 -- 172.21.15.131:0/446432386 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f4c8c092aa0 con 0x7f4c940fd7a0 2023-06-23T01:03:39.443 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:39.440+0000 7f4c99728700 1 -- 172.21.15.131:0/446432386 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c94115f30 con 0x7f4c7c09b6f0 2023-06-23T01:03:39.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:39 smithi092 ceph-mon[85670]: osdmap e14: 2 total, 2 up, 2 in 2023-06-23T01:03:39.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:03:39.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:03:39.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:39.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:39 smithi131 ceph-mon[86495]: osdmap e14: 2 total, 2 up, 2 in 2023-06-23T01:03:39.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:03:39.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:03:39.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:40.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:39 smithi046 ceph-mon[84949]: osdmap e14: 2 total, 2 up, 2 in 2023-06-23T01:03:40.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:03:40.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:03:40.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:40.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:40 smithi092 ceph-mon[85670]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:03:40.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:40 smithi092 ceph-mon[85670]: pgmap v56: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:40.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:40 smithi131 ceph-mon[86495]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:03:40.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:40 smithi131 ceph-mon[86495]: pgmap v56: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:41.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:40 smithi046 ceph-mon[84949]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:03:41.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:40 smithi046 ceph-mon[84949]: pgmap v56: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:42.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:41 smithi092 ceph-mon[85670]: from='client.? 172.21.15.131:0/3462314726' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec7b0c60-88ea-4c2b-a34c-96d188e02014"}]: dispatch 2023-06-23T01:03:42.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:41 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec7b0c60-88ea-4c2b-a34c-96d188e02014"}]: dispatch 2023-06-23T01:03:42.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:41 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec7b0c60-88ea-4c2b-a34c-96d188e02014"}]': finished 2023-06-23T01:03:42.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:41 smithi092 ceph-mon[85670]: osdmap e15: 3 total, 2 up, 3 in 2023-06-23T01:03:42.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:41 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:03:42.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:41 smithi131 ceph-mon[86495]: from='client.? 172.21.15.131:0/3462314726' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec7b0c60-88ea-4c2b-a34c-96d188e02014"}]: dispatch 2023-06-23T01:03:42.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:41 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec7b0c60-88ea-4c2b-a34c-96d188e02014"}]: dispatch 2023-06-23T01:03:42.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:41 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec7b0c60-88ea-4c2b-a34c-96d188e02014"}]': finished 2023-06-23T01:03:42.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:41 smithi131 ceph-mon[86495]: osdmap e15: 3 total, 2 up, 3 in 2023-06-23T01:03:42.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:41 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:03:42.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:41 smithi046 ceph-mon[84949]: from='client.? 172.21.15.131:0/3462314726' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec7b0c60-88ea-4c2b-a34c-96d188e02014"}]: dispatch 2023-06-23T01:03:42.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:41 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec7b0c60-88ea-4c2b-a34c-96d188e02014"}]: dispatch 2023-06-23T01:03:42.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:41 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec7b0c60-88ea-4c2b-a34c-96d188e02014"}]': finished 2023-06-23T01:03:42.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:41 smithi046 ceph-mon[84949]: osdmap e15: 3 total, 2 up, 3 in 2023-06-23T01:03:42.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:41 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:03:43.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:42 smithi092 ceph-mon[85670]: pgmap v58: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:43.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:42 smithi092 ceph-mon[85670]: from='client.? 172.21.15.131:0/3915487903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:03:43.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:42 smithi131 ceph-mon[86495]: pgmap v58: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:43.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:42 smithi131 ceph-mon[86495]: from='client.? 172.21.15.131:0/3915487903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:03:43.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:42 smithi046 ceph-mon[84949]: pgmap v58: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:43.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:42 smithi046 ceph-mon[84949]: from='client.? 172.21.15.131:0/3915487903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:03:45.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:44 smithi092 ceph-mon[85670]: pgmap v59: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:45.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:44 smithi131 ceph-mon[86495]: pgmap v59: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:45.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:44 smithi046 ceph-mon[84949]: pgmap v59: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:46.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:46 smithi131 ceph-mon[86495]: pgmap v60: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:47.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:46 smithi092 ceph-mon[85670]: pgmap v60: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:47.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:46 smithi046 ceph-mon[84949]: pgmap v60: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:48.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:48 smithi131 ceph-mon[86495]: pgmap v61: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:48.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:48 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-06-23T01:03:48.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:48 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:48.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:48 smithi046 ceph-mon[84949]: pgmap v61: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:48.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:48 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-06-23T01:03:48.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:48 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:48.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:48 smithi092 ceph-mon[85670]: pgmap v61: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:48.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:48 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-06-23T01:03:48.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:48 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:49.726 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:49 smithi131 ceph-mon[86495]: Deploying daemon osd.2 on smithi131 2023-06-23T01:03:49.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:49 smithi046 ceph-mon[84949]: Deploying daemon osd.2 on smithi131 2023-06-23T01:03:49.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:49 smithi092 ceph-mon[85670]: Deploying daemon osd.2 on smithi131 2023-06-23T01:03:50.662 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:50 smithi131 ceph-mon[86495]: pgmap v62: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:50.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:50 smithi046 ceph-mon[84949]: pgmap v62: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:50.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:50 smithi092 ceph-mon[85670]: pgmap v62: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:53.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:52 smithi092 ceph-mon[85670]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:53.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:52 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:53.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:52 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:53.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:52 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:53.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:52 smithi131 ceph-mon[86495]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:53.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:52 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:53.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:52 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:53.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:52 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:53.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:52 smithi046 ceph-mon[84949]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:53.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:52 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:53.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:52 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:53.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:52 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:55.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:54 smithi092 ceph-mon[85670]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:55.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:54 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:55.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:54 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:55.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:54 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:55.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:54 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:55.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:54 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:55.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:54 smithi131 ceph-mon[86495]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:55.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:54 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:55.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:54 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:55.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:54 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:55.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:54 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:55.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:54 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:55.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:54 smithi046 ceph-mon[84949]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:55.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:54 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:55.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:54 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:55.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:54 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:55.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:54 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:55.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:54 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:55.503 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.500+0000 7f4c7b7fe700 1 -- 172.21.15.131:0/446432386 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4c94115f30 con 0x7f4c7c09b6f0 2023-06-23T01:03:55.504 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 2 on host 'smithi131' 2023-06-23T01:03:55.506 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.502+0000 7f4c99728700 1 -- 172.21.15.131:0/446432386 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4c7c09b6f0 msgr2=0x7f4c7c09dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:55.506 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.502+0000 7f4c99728700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4c7c09b6f0 0x7f4c7c09dba0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009bf0 tx=0x7f4c840091a0 comp rx=0 tx=0).stop 2023-06-23T01:03:55.506 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.502+0000 7f4c99728700 1 -- 172.21.15.131:0/446432386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4c940fd7a0 msgr2=0x7f4c94114d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:03:55.506 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.503+0000 7f4c99728700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4c940fd7a0 0x7f4c94114d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009fd0 tx=0x7f4c8c004d10 comp rx=0 tx=0).stop 2023-06-23T01:03:55.506 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.503+0000 7f4c99728700 1 -- 172.21.15.131:0/446432386 shutdown_connections 2023-06-23T01:03:55.506 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.503+0000 7f4c99728700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4c940fd7a0 0x7f4c94114d80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:55.506 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.503+0000 7f4c99728700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c940fbb40 0x7f4c9410d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:55.507 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.503+0000 7f4c99728700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4c7c09b6f0 0x7f4c7c09dba0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:55.507 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.503+0000 7f4c99728700 1 --2- 172.21.15.131:0/446432386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4c940fcdf0 0x7f4c9410dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:03:55.507 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.503+0000 7f4c99728700 1 -- 172.21.15.131:0/446432386 >> 172.21.15.131:0/446432386 conn(0x7f4c940f72a0 msgr2=0x7f4c94065ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:03:55.507 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.503+0000 7f4c99728700 1 -- 172.21.15.131:0/446432386 shutdown_connections 2023-06-23T01:03:55.507 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:03:55.503+0000 7f4c99728700 1 -- 172.21.15.131:0/446432386 wait complete. 2023-06-23T01:03:55.933 DEBUG:teuthology.orchestra.run.smithi131:osd.2> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.2.service 2023-06-23T01:03:55.935 INFO:tasks.cephadm:Deploying osd.3 on smithi046 with /dev/vg_nvme/lv_3... 2023-06-23T01:03:55.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-06-23T01:03:55.964 INFO:journalctl@ceph.osd.2.smithi131.stdout:-- Logs begin at Fri 2023-06-23 00:44:02 UTC. -- 2023-06-23T01:03:56.548 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:03:56.635 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:56.635 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:56.635 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:56.635 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:56 smithi131 ceph-mon[86495]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:56.748 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:56.748 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:56.748 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:56.748 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:56 smithi046 ceph-mon[84949]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:56.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:56.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:56.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:03:56.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:56 smithi092 ceph-mon[85670]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:56.981 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:03:56 smithi131 conmon[94162]: 2023-06-23T01:03:56.630+0000 7f2bca644540 -1 osd.2 0 log_to_monitors true 2023-06-23T01:03:57.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:57 smithi046 ceph-mon[84949]: from='osd.2 [v2:172.21.15.131:6800/1787957441,v1:172.21.15.131:6801/1787957441]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-23T01:03:57.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:57 smithi046 ceph-mon[84949]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-23T01:03:57.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:57 smithi131 ceph-mon[86495]: from='osd.2 [v2:172.21.15.131:6800/1787957441,v1:172.21.15.131:6801/1787957441]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-23T01:03:57.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:57 smithi131 ceph-mon[86495]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-23T01:03:57.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:57 smithi092 ceph-mon[85670]: from='osd.2 [v2:172.21.15.131:6800/1787957441,v1:172.21.15.131:6801/1787957441]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-23T01:03:57.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:57 smithi092 ceph-mon[85670]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-06-23T01:03:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-06-23T01:03:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-06-23T01:03:58.304 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2023-06-23T01:03:58.305 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2023-06-23T01:03:58.305 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00864481 s, 1.2 GB/s 2023-06-23T01:03:58.305 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2023-06-23T01:03:58.677 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-06-23T01:03:58.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: osdmap e16: 3 total, 2 up, 3 in 2023-06-23T01:03:58.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:03:58.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: from='osd.2 [v2:172.21.15.131:6800/1787957441,v1:172.21.15.131:6801/1787957441]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:03:58.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:03:58.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:58.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi131 2023-06-23T01:03:58.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:58.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:58.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:03:58.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi131 to 17168M 2023-06-23T01:03:58.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:58.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:58.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:58.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:58.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi046:vg_nvme/lv_3 2023-06-23T01:03:58.731 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:03:58 smithi131 conmon[94162]: 2023-06-23T01:03:58.477+0000 7f2bc262c700 -1 osd.2 0 waiting for initial osdmap 2023-06-23T01:03:58.732 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:03:58 smithi131 conmon[94162]: 2023-06-23T01:03:58.481+0000 7f2bbac22700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:03:58.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-06-23T01:03:58.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: osdmap e16: 3 total, 2 up, 3 in 2023-06-23T01:03:58.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:03:58.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: from='osd.2 [v2:172.21.15.131:6800/1787957441,v1:172.21.15.131:6801/1787957441]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:03:58.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:03:58.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:58.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi131 2023-06-23T01:03:58.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:58.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:58.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:03:58.734 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi131 to 17168M 2023-06-23T01:03:58.734 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:58.734 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:58.734 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:58.734 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:58.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-06-23T01:03:58.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: osdmap e16: 3 total, 2 up, 3 in 2023-06-23T01:03:58.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:03:58.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: from='osd.2 [v2:172.21.15.131:6800/1787957441,v1:172.21.15.131:6801/1787957441]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:03:58.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:03:58.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:03:58.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi131 2023-06-23T01:03:58.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:58.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:58.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:03:58.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi131 to 17168M 2023-06-23T01:03:58.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:58.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:03:58.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:03:58.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:03:59.293 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:03:59.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:59 smithi131 ceph-mon[86495]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:03:59.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:59 smithi131 ceph-mon[86495]: osdmap e17: 3 total, 2 up, 3 in 2023-06-23T01:03:59.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:03:59.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:03:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:03:59.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:59 smithi046 ceph-mon[84949]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:03:59.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:59 smithi046 ceph-mon[84949]: osdmap e17: 3 total, 2 up, 3 in 2023-06-23T01:03:59.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:03:59.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:03:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:03:59.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:59 smithi092 ceph-mon[85670]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:03:59.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:59 smithi092 ceph-mon[85670]: osdmap e17: 3 total, 2 up, 3 in 2023-06-23T01:03:59.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:03:59.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:03:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:04:00.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.511+0000 7fdc95839700 1 -- 172.21.15.46:0/3536265913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc900fdfa0 msgr2=0x7fdc900fe3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:00.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.511+0000 7fdc95839700 1 --2- 172.21.15.46:0/3536265913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc900fdfa0 0x7fdc900fe3f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdc78004a20 tx=0x7fdc78009e30 comp rx=0 tx=0).stop 2023-06-23T01:04:00.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.511+0000 7fdc95839700 1 -- 172.21.15.46:0/3536265913 shutdown_connections 2023-06-23T01:04:00.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.511+0000 7fdc95839700 1 --2- 172.21.15.46:0/3536265913 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc900fe930 0x7fdc901051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:00.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.511+0000 7fdc95839700 1 --2- 172.21.15.46:0/3536265913 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc900faf10 0x7fdc900fb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:00.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.511+0000 7fdc95839700 1 --2- 172.21.15.46:0/3536265913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc900fdfa0 0x7fdc900fe3f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:00.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.511+0000 7fdc95839700 1 -- 172.21.15.46:0/3536265913 >> 172.21.15.46:0/3536265913 conn(0x7fdc900f6a50 msgr2=0x7fdc900f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:04:00.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.511+0000 7fdc95839700 1 -- 172.21.15.46:0/3536265913 shutdown_connections 2023-06-23T01:04:00.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.511+0000 7fdc95839700 1 -- 172.21.15.46:0/3536265913 wait complete. 2023-06-23T01:04:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc95839700 1 Processor -- start 2023-06-23T01:04:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc95839700 1 -- start start 2023-06-23T01:04:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc95839700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc900faf10 0x7fdc90113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc95839700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc900fdfa0 0x7fdc90114310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc95839700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc900fe930 0x7fdc9011b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:00.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc95839700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9011b9b0 con 0x7fdc900faf10 2023-06-23T01:04:00.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc95839700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9011bb20 con 0x7fdc900fdfa0 2023-06-23T01:04:00.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc95839700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc9011bce0 con 0x7fdc900fe930 2023-06-23T01:04:00.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc8effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc900faf10 0x7fdc90113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:00.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc8effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc900faf10 0x7fdc90113dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45466/0 (socket says 172.21.15.46:45466) 2023-06-23T01:04:00.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc8e7fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc900fdfa0 0x7fdc90114310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:00.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc8effd700 1 -- 172.21.15.46:0/2728251458 learned_addr learned my addr 172.21.15.46:0/2728251458 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:04:00.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.512+0000 7fdc8f7fe700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc900fe930 0x7fdc9011b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:00.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.513+0000 7fdc8effd700 1 -- 172.21.15.46:0/2728251458 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc900fe930 msgr2=0x7fdc9011b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:00.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.513+0000 7fdc8effd700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc900fe930 0x7fdc9011b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:00.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.513+0000 7fdc8effd700 1 -- 172.21.15.46:0/2728251458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc900fdfa0 msgr2=0x7fdc90114310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:00.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.513+0000 7fdc8effd700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc900fdfa0 0x7fdc90114310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:00.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.513+0000 7fdc8effd700 1 -- 172.21.15.46:0/2728251458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc78005040 con 0x7fdc900faf10 2023-06-23T01:04:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.513+0000 7fdc8effd700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc900faf10 0x7fdc90113dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800cb90 tx=0x7fdc8800cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:04:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.513+0000 7fdc877fe700 1 -- 172.21.15.46:0/2728251458 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8800a7a0 con 0x7fdc900faf10 2023-06-23T01:04:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.513+0000 7fdc877fe700 1 -- 172.21.15.46:0/2728251458 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdc8800ada0 con 0x7fdc900faf10 2023-06-23T01:04:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.513+0000 7fdc95839700 1 -- 172.21.15.46:0/2728251458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc9011bfc0 con 0x7fdc900faf10 2023-06-23T01:04:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.513+0000 7fdc877fe700 1 -- 172.21.15.46:0/2728251458 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88012c90 con 0x7fdc900faf10 2023-06-23T01:04:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.513+0000 7fdc95839700 1 -- 172.21.15.46:0/2728251458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc9011c450 con 0x7fdc900faf10 2023-06-23T01:04:00.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.514+0000 7fdc95839700 1 -- 172.21.15.46:0/2728251458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc90102300 con 0x7fdc900faf10 2023-06-23T01:04:00.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.516+0000 7fdc877fe700 1 -- 172.21.15.46:0/2728251458 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 15) v1 ==== 129145+0+0 (secure 0 0 0) 0x7fdc88003020 con 0x7fdc900faf10 2023-06-23T01:04:00.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.516+0000 7fdc877fe700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fdc7c09b670 0x7fdc7c09db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:00.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.516+0000 7fdc8e7fc700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fdc7c09b670 0x7fdc7c09db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:00.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.517+0000 7fdc877fe700 1 -- 172.21.15.46:0/2728251458 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3222+0+0 (secure 0 0 0) 0x7fdc880c4280 con 0x7fdc900faf10 2023-06-23T01:04:00.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.517+0000 7fdc8e7fc700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fdc7c09b670 0x7fdc7c09db20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc78004f90 tx=0x7fdc78009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:04:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.520+0000 7fdc877fe700 1 -- 172.21.15.46:0/2728251458 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fdc88090f00 con 0x7fdc900faf10 2023-06-23T01:04:00.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:00 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:04:00.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:00 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:04:00.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:00 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:04:00.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:00 smithi046 ceph-mon[84949]: osd.2 [v2:172.21.15.131:6800/1787957441,v1:172.21.15.131:6801/1787957441] boot 2023-06-23T01:04:00.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:00 smithi046 ceph-mon[84949]: osdmap e18: 3 total, 3 up, 3 in 2023-06-23T01:04:00.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:00 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:04:00.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:00 smithi046 ceph-mon[84949]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:04:00.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:00 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-06-23T01:04:00.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:00.721+0000 7fdc95839700 1 -- 172.21.15.46:0/2728251458 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc90108510 con 0x7fdc7c09b670 2023-06-23T01:04:00.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:00 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:04:00.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:00 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:04:00.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:00 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:04:00.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:00 smithi131 ceph-mon[86495]: osd.2 [v2:172.21.15.131:6800/1787957441,v1:172.21.15.131:6801/1787957441] boot 2023-06-23T01:04:00.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:00 smithi131 ceph-mon[86495]: osdmap e18: 3 total, 3 up, 3 in 2023-06-23T01:04:00.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:00 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:04:00.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:00 smithi131 ceph-mon[86495]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:04:00.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:00 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-06-23T01:04:00.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:00 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:04:00.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:00 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:04:00.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:00 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:04:00.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:00 smithi092 ceph-mon[85670]: osd.2 [v2:172.21.15.131:6800/1787957441,v1:172.21.15.131:6801/1787957441] boot 2023-06-23T01:04:00.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:00 smithi092 ceph-mon[85670]: osdmap e18: 3 total, 3 up, 3 in 2023-06-23T01:04:00.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:00 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-06-23T01:04:00.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:00 smithi092 ceph-mon[85670]: pgmap v70: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-06-23T01:04:00.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:00 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-06-23T01:04:01.738 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:01 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-06-23T01:04:01.739 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:01 smithi046 ceph-mon[84949]: osdmap e19: 3 total, 3 up, 3 in 2023-06-23T01:04:01.739 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:01 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-06-23T01:04:01.739 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:01 smithi046 ceph-mon[84949]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:04:01.740 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:01 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:04:01.740 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:01 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:04:01.740 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:01 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:01.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:01 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-06-23T01:04:01.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:01 smithi092 ceph-mon[85670]: osdmap e19: 3 total, 3 up, 3 in 2023-06-23T01:04:01.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:01 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-06-23T01:04:01.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:01 smithi092 ceph-mon[85670]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:04:01.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:01 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:04:01.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:01 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:04:01.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:01 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:01.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:01 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-06-23T01:04:01.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:01 smithi131 ceph-mon[86495]: osdmap e19: 3 total, 3 up, 3 in 2023-06-23T01:04:01.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:01 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-06-23T01:04:01.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:01 smithi131 ceph-mon[86495]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:04:01.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:01 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:04:01.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:01 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:04:01.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:01 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:02.318 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:04:02 smithi046 sudo[105858]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-23T01:04:02.318 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:04:02 smithi046 sudo[105858]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:02.318 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:04:02 smithi046 sudo[105858]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:02.318 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:04:02 smithi046 sudo[105858]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:02.753 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-06-23T01:04:02.754 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:02 smithi092 ceph-mon[85670]: osdmap e20: 3 total, 3 up, 3 in 2023-06-23T01:04:02.754 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:02 smithi092 ceph-mon[85670]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:02.754 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:04:02 smithi092 sudo[94835]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-23T01:04:02.754 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:04:02 smithi092 sudo[94835]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:02.755 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:04:02 smithi092 sudo[94835]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:02.755 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:04:02 smithi092 sudo[94835]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:02.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-06-23T01:04:02.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:02 smithi046 ceph-mon[84949]: osdmap e20: 3 total, 3 up, 3 in 2023-06-23T01:04:02.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:02 smithi046 ceph-mon[84949]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:02.819 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:04:02 smithi046 sudo[105939]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-23T01:04:02.819 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:04:02 smithi046 sudo[105939]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:02.819 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:04:02 smithi046 sudo[105939]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:02.819 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:04:02 smithi046 sudo[105939]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:02.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-06-23T01:04:02.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:02 smithi131 ceph-mon[86495]: osdmap e20: 3 total, 3 up, 3 in 2023-06-23T01:04:02.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:02 smithi131 ceph-mon[86495]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:03.139 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:04:02 smithi092 sudo[94838]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-23T01:04:03.140 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:04:02 smithi092 sudo[94838]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:03.140 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:04:02 smithi092 sudo[94838]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:03.140 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:04:02 smithi092 sudo[94838]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:03.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:03.402+0000 7fdc877fe700 1 -- 172.21.15.46:0/2728251458 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fdc880912d0 con 0x7fdc900faf10 2023-06-23T01:04:03.489 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:04:03 smithi131 sudo[96888]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-23T01:04:03.489 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:04:03 smithi131 sudo[96888]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:03.490 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:04:03 smithi131 sudo[96888]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:03.490 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:04:03 smithi131 sudo[96888]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:03.490 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:04:03 smithi131 sudo[96891]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-23T01:04:03.490 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:04:03 smithi131 sudo[96891]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:03.491 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:04:03 smithi131 sudo[96891]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:03.491 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:04:03 smithi131 sudo[96891]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:03.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 ceph-mon[84949]: osdmap e21: 3 total, 3 up, 3 in 2023-06-23T01:04:03.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:03.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:03.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2274237574' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32670a23-3286-4538-9628-38d763429cbd"}]: dispatch 2023-06-23T01:04:03.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2274237574' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32670a23-3286-4538-9628-38d763429cbd"}]': finished 2023-06-23T01:04:03.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 ceph-mon[84949]: osdmap e22: 4 total, 3 up, 4 in 2023-06-23T01:04:03.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:03.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 ceph-mon[84949]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-06-23T01:04:03.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:03 smithi092 ceph-mon[85670]: osdmap e21: 3 total, 3 up, 3 in 2023-06-23T01:04:03.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:03 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:03.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:03 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:03.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:03 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2274237574' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32670a23-3286-4538-9628-38d763429cbd"}]: dispatch 2023-06-23T01:04:03.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:03 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2274237574' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32670a23-3286-4538-9628-38d763429cbd"}]': finished 2023-06-23T01:04:03.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:03 smithi092 ceph-mon[85670]: osdmap e22: 4 total, 3 up, 4 in 2023-06-23T01:04:03.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:03 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:03.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:03 smithi092 ceph-mon[85670]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-06-23T01:04:03.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:03 smithi131 ceph-mon[86495]: osdmap e21: 3 total, 3 up, 3 in 2023-06-23T01:04:03.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:03 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:03.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:03 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:03.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:03 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2274237574' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32670a23-3286-4538-9628-38d763429cbd"}]: dispatch 2023-06-23T01:04:03.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:03 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2274237574' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32670a23-3286-4538-9628-38d763429cbd"}]': finished 2023-06-23T01:04:03.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:03 smithi131 ceph-mon[86495]: osdmap e22: 4 total, 3 up, 4 in 2023-06-23T01:04:03.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:03 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:03.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:03 smithi131 ceph-mon[86495]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-06-23T01:04:04.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 sudo[106013]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-23T01:04:04.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 sudo[106013]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:04.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 sudo[106013]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:04.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:03 smithi046 sudo[106013]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:04.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:04 smithi046 sudo[106016]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-23T01:04:04.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:04 smithi046 sudo[106016]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:04.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:04 smithi046 sudo[106016]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:04.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:04 smithi046 sudo[106016]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:04.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:04 smithi131 sudo[96894]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-23T01:04:04.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:04 smithi131 sudo[96894]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:04.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:04 smithi131 sudo[96894]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:04.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:04 smithi131 sudo[96894]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:04.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:04 smithi131 sudo[96897]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-23T01:04:04.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:04 smithi131 sudo[96897]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:04.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:04 smithi131 sudo[96897]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:04.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:04 smithi131 sudo[96897]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:05.359 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 sudo[94841]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-06-23T01:04:05.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 sudo[94841]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:05.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 sudo[94841]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:05.360 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 sudo[94841]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:05.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 sudo[94844]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-06-23T01:04:05.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 sudo[94844]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-06-23T01:04:05.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 sudo[94844]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-06-23T01:04:05.361 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 sudo[94844]: pam_unix(sudo:session): session closed for user root 2023-06-23T01:04:05.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:05.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 ceph-mon[85670]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-23T01:04:05.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 ceph-mon[85670]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:05.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 ceph-mon[85670]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-23T01:04:05.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:04:05.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/3228621572' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:04:05.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:04:05.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:04:05.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:04:05.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:04:05.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:05 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:04:05.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:05 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:05.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:05 smithi131 ceph-mon[86495]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-23T01:04:05.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:05 smithi131 ceph-mon[86495]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:05.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:05 smithi131 ceph-mon[86495]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-23T01:04:05.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:05 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:04:05.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:05 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/3228621572' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:04:05.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:05 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:04:05.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:05 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:04:05.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:05 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:04:05.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:05 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:04:05.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:05 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:04:05.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:05 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:05.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:05 smithi046 ceph-mon[84949]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-23T01:04:05.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:05 smithi046 ceph-mon[84949]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:05.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:05 smithi046 ceph-mon[84949]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-23T01:04:05.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:05 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:04:05.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:05 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3228621572' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:04:05.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:05 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:04:05.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:05 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:04:05.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:05 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:04:05.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:05 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:04:05.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:05 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:04:06.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:06 smithi092 ceph-mon[85670]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-23T01:04:06.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:06 smithi092 ceph-mon[85670]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-23T01:04:06.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:06 smithi092 ceph-mon[85670]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-23T01:04:06.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:06 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:04:06.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:06 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:04:06.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:06 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:04:06.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:06 smithi092 ceph-mon[85670]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-23T01:04:06.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:06 smithi092 ceph-mon[85670]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:06.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:06 smithi131 ceph-mon[86495]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-23T01:04:06.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:06 smithi131 ceph-mon[86495]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-23T01:04:06.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:06 smithi131 ceph-mon[86495]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-23T01:04:06.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:06 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:04:06.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:06 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:04:06.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:06 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:04:06.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:06 smithi131 ceph-mon[86495]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-23T01:04:06.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:06 smithi131 ceph-mon[86495]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:06.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:06 smithi046 ceph-mon[84949]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-23T01:04:06.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:06 smithi046 ceph-mon[84949]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-23T01:04:06.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:06 smithi046 ceph-mon[84949]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-06-23T01:04:06.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-06-23T01:04:06.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-06-23T01:04:06.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-06-23T01:04:06.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:06 smithi046 ceph-mon[84949]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-06-23T01:04:06.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:06 smithi046 ceph-mon[84949]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:08.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:08 smithi131 ceph-mon[86495]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:08.771 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:08 smithi046 ceph-mon[84949]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:08.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:08 smithi092 ceph-mon[85670]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:11.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:10 smithi092 ceph-mon[85670]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-06-23T01:04:11.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:10 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-06-23T01:04:11.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:10 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:11.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:10 smithi131 ceph-mon[86495]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-06-23T01:04:11.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:10 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-06-23T01:04:11.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:10 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:11.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:10 smithi046 ceph-mon[84949]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-06-23T01:04:11.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:10 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-06-23T01:04:11.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:10 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:12.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:11 smithi092 ceph-mon[85670]: Deploying daemon osd.3 on smithi046 2023-06-23T01:04:12.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:11 smithi131 ceph-mon[86495]: Deploying daemon osd.3 on smithi046 2023-06-23T01:04:12.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:11 smithi046 ceph-mon[84949]: Deploying daemon osd.3 on smithi046 2023-06-23T01:04:13.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:12 smithi046 ceph-mon[84949]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-06-23T01:04:13.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:12 smithi092 ceph-mon[85670]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-06-23T01:04:13.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:12 smithi131 ceph-mon[86495]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-06-23T01:04:15.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:14 smithi046 ceph-mon[84949]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-06-23T01:04:15.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:14 smithi092 ceph-mon[85670]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-06-23T01:04:15.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:14 smithi131 ceph-mon[86495]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-06-23T01:04:16.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:16.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:16.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:16.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:16 smithi046 ceph-mon[84949]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-06-23T01:04:16.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:16.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:16.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:16.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:16 smithi131 ceph-mon[86495]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-06-23T01:04:16.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:16.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:16.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:16.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:16 smithi092 ceph-mon[85670]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-06-23T01:04:18.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.322+0000 7fdc877fe700 1 -- 172.21.15.46:0/2728251458 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdc7800bd40 con 0x7fdc7c09b670 2023-06-23T01:04:18.325 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 3 on host 'smithi046' 2023-06-23T01:04:18.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 -- 172.21.15.46:0/2728251458 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fdc7c09b670 msgr2=0x7fdc7c09db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fdc7c09b670 0x7fdc7c09db20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc78004f90 tx=0x7fdc78009e30 comp rx=0 tx=0).stop 2023-06-23T01:04:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 -- 172.21.15.46:0/2728251458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc900faf10 msgr2=0x7fdc90113dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc900faf10 0x7fdc90113dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800cb90 tx=0x7fdc8800cea0 comp rx=0 tx=0).stop 2023-06-23T01:04:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 -- 172.21.15.46:0/2728251458 shutdown_connections 2023-06-23T01:04:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fdc900fe930 0x7fdc9011b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc900faf10 0x7fdc90113dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:18.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fdc7c09b670 0x7fdc7c09db20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 --2- 172.21.15.46:0/2728251458 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc900fdfa0 0x7fdc90114310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 -- 172.21.15.46:0/2728251458 >> 172.21.15.46:0/2728251458 conn(0x7fdc900f6a50 msgr2=0x7fdc900ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:04:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 -- 172.21.15.46:0/2728251458 shutdown_connections 2023-06-23T01:04:18.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:04:18.324+0000 7fdc95839700 1 -- 172.21.15.46:0/2728251458 wait complete. 2023-06-23T01:04:18.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:18 smithi046 ceph-mon[84949]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-06-23T01:04:18.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:18.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:18.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:18.669 DEBUG:teuthology.orchestra.run.smithi046:osd.3> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.3.service 2023-06-23T01:04:18.671 INFO:tasks.cephadm:Deploying osd.4 on smithi092 with /dev/vg_nvme/lv_3... 2023-06-23T01:04:18.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-06-23T01:04:18.705 INFO:journalctl@ceph.osd.3.smithi046.stdout:-- Logs begin at Fri 2023-06-23 00:44:15 UTC. -- 2023-06-23T01:04:18.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:18 smithi131 ceph-mon[86495]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-06-23T01:04:18.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:18.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:18.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:18.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:18 smithi092 ceph-mon[85670]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-06-23T01:04:18.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:18.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:18.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:18.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:19.259 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-23T01:04:19.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:19 smithi131 ceph-mon[86495]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-06-23T01:04:19.784 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:19 smithi046 ceph-mon[84949]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-06-23T01:04:19.785 INFO:journalctl@ceph.osd.3.smithi046.stdout:Jun 23 01:04:19 smithi046 conmon[108845]: 2023-06-23T01:04:19.414+0000 7f1440ae1540 -1 osd.3 0 log_to_monitors true 2023-06-23T01:04:19.862 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:19 smithi092 ceph-mon[85670]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-06-23T01:04:20.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:20 smithi092 ceph-mon[85670]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-06-23T01:04:20.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:20 smithi092 ceph-mon[85670]: osdmap e23: 4 total, 3 up, 4 in 2023-06-23T01:04:20.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:20 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:20.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:20 smithi092 ceph-mon[85670]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:04:20.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:20 smithi092 ceph-mon[85670]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:20.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:20 smithi131 ceph-mon[86495]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-06-23T01:04:20.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:20 smithi131 ceph-mon[86495]: osdmap e23: 4 total, 3 up, 4 in 2023-06-23T01:04:20.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:20 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:20.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:20 smithi131 ceph-mon[86495]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:04:20.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:20 smithi131 ceph-mon[86495]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:20.760 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-06-23T01:04:20.761 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-06-23T01:04:20.761 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2023-06-23T01:04:20.761 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2023-06-23T01:04:20.761 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113162 s, 927 MB/s 2023-06-23T01:04:20.762 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2023-06-23T01:04:20.818 INFO:journalctl@ceph.osd.3.smithi046.stdout:Jun 23 01:04:20 smithi046 conmon[108845]: 2023-06-23T01:04:20.470+0000 7f14372c6700 -1 osd.3 0 waiting for initial osdmap 2023-06-23T01:04:20.818 INFO:journalctl@ceph.osd.3.smithi046.stdout:Jun 23 01:04:20 smithi046 conmon[108845]: 2023-06-23T01:04:20.479+0000 7f14318c0700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:04:20.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:20 smithi046 ceph-mon[84949]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-06-23T01:04:20.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:20 smithi046 ceph-mon[84949]: osdmap e23: 4 total, 3 up, 4 in 2023-06-23T01:04:20.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:20 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:20.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:20 smithi046 ceph-mon[84949]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:04:20.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:20 smithi046 ceph-mon[84949]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:21.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi092:vg_nvme/lv_3 2023-06-23T01:04:21.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:04:21.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: osdmap e24: 4 total, 3 up, 4 in 2023-06-23T01:04:21.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:21.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:21.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:21.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:21.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:21.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:21.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:21.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:21.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:21.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:21.781 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-23T01:04:21.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:04:21.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: osdmap e24: 4 total, 3 up, 4 in 2023-06-23T01:04:21.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:21.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:21.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:21.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:21.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:21.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:21.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:21.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:21.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:21.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:21.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: from='osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:04:21.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: osdmap e24: 4 total, 3 up, 4 in 2023-06-23T01:04:21.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:21.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:21.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:21.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:21.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:21.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:21.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:21.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:21.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:21.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:22.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:22 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:04:22.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:22 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:04:22.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:22 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi046 2023-06-23T01:04:22.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:22 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi046 to 8586M 2023-06-23T01:04:22.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:22 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:22.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:22 smithi092 ceph-mon[85670]: osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444] boot 2023-06-23T01:04:22.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:22 smithi092 ceph-mon[85670]: osdmap e25: 4 total, 4 up, 4 in 2023-06-23T01:04:22.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:22 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:22.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:22 smithi092 ceph-mon[85670]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:22.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:22 smithi092 ceph-mon[85670]: osdmap e26: 4 total, 4 up, 4 in 2023-06-23T01:04:22.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:22 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:04:22.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:22 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:04:22.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:22 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi046 2023-06-23T01:04:22.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:22 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi046 to 8586M 2023-06-23T01:04:22.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:22 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:22.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:22 smithi131 ceph-mon[86495]: osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444] boot 2023-06-23T01:04:22.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:22 smithi131 ceph-mon[86495]: osdmap e25: 4 total, 4 up, 4 in 2023-06-23T01:04:22.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:22 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:22.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:22 smithi131 ceph-mon[86495]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:22.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:22 smithi131 ceph-mon[86495]: osdmap e26: 4 total, 4 up, 4 in 2023-06-23T01:04:22.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:22 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:04:22.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:22 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:04:22.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:22 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi046 2023-06-23T01:04:22.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:22 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi046 to 8586M 2023-06-23T01:04:22.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:22 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:22.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:22 smithi046 ceph-mon[84949]: osd.3 [v2:172.21.15.46:6810/338690444,v1:172.21.15.46:6811/338690444] boot 2023-06-23T01:04:22.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:22 smithi046 ceph-mon[84949]: osdmap e25: 4 total, 4 up, 4 in 2023-06-23T01:04:22.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:22 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-06-23T01:04:22.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:22 smithi046 ceph-mon[84949]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-06-23T01:04:22.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:22 smithi046 ceph-mon[84949]: osdmap e26: 4 total, 4 up, 4 in 2023-06-23T01:04:22.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 -- 172.21.15.92:0/2680093923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5ec0fd1d0 msgr2=0x7fc5ec0fd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:22.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 --2- 172.21.15.92:0/2680093923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5ec0fd1d0 0x7fc5ec0fd5e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009d60 tx=0x7fc5e8004d10 comp rx=0 tx=0).stop 2023-06-23T01:04:22.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 -- 172.21.15.92:0/2680093923 shutdown_connections 2023-06-23T01:04:22.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 --2- 172.21.15.92:0/2680093923 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5ec0fd1d0 0x7fc5ec0fd5e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:22.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 --2- 172.21.15.92:0/2680093923 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc5ec0fc800 0x7fc5ec1040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:22.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 --2- 172.21.15.92:0/2680093923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0fbe50 0x7fc5ec0fc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:22.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 -- 172.21.15.92:0/2680093923 >> 172.21.15.92:0/2680093923 conn(0x7fc5ec0f6a50 msgr2=0x7fc5ec0f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:04:22.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 -- 172.21.15.92:0/2680093923 shutdown_connections 2023-06-23T01:04:22.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 -- 172.21.15.92:0/2680093923 wait complete. 2023-06-23T01:04:22.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 Processor -- start 2023-06-23T01:04:22.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 -- start start 2023-06-23T01:04:22.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.979+0000 7fc5f380b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0fbe50 0x7fc5ec106090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:22.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f380b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc5ec0fc800 0x7fc5ec104740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:22.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f380b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5ec0fd1d0 0x7fc5ec104c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f380b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec105420 con 0x7fc5ec0fbe50 2023-06-23T01:04:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f380b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec105590 con 0x7fc5ec0fd1d0 2023-06-23T01:04:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f380b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec117a60 con 0x7fc5ec0fc800 2023-06-23T01:04:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f1da8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5ec0fd1d0 0x7fc5ec104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f15a7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0fbe50 0x7fc5ec106090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f1da8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5ec0fd1d0 0x7fc5ec104c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:40230/0 (socket says 172.21.15.92:40230) 2023-06-23T01:04:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f1da8700 1 -- 172.21.15.92:0/1862481945 learned_addr learned my addr 172.21.15.92:0/1862481945 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:04:22.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f0da6700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc5ec0fc800 0x7fc5ec104740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f1da8700 1 -- 172.21.15.92:0/1862481945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc5ec0fc800 msgr2=0x7fc5ec104740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f1da8700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc5ec0fc800 0x7fc5ec104740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f1da8700 1 -- 172.21.15.92:0/1862481945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0fbe50 msgr2=0x7fc5ec106090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f1da8700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0fbe50 0x7fc5ec106090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f1da8700 1 -- 172.21.15.92:0/1862481945 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e8009940 con 0x7fc5ec0fd1d0 2023-06-23T01:04:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f0da6700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc5ec0fc800 0x7fc5ec104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:04:22.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f1da8700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5ec0fd1d0 0x7fc5ec104c80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400b810 tx=0x7fc5e40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:04:22.989 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5e1ffb700 1 -- 172.21.15.92:0/1862481945 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e400f510 con 0x7fc5ec0fd1d0 2023-06-23T01:04:22.989 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5e1ffb700 1 -- 172.21.15.92:0/1862481945 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc5e400fb10 con 0x7fc5ec0fd1d0 2023-06-23T01:04:22.989 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5f380b700 1 -- 172.21.15.92:0/1862481945 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec117cc0 con 0x7fc5ec0fd1d0 2023-06-23T01:04:22.989 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.980+0000 7fc5e1ffb700 1 -- 172.21.15.92:0/1862481945 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4011b50 con 0x7fc5ec0fd1d0 2023-06-23T01:04:22.989 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.981+0000 7fc5f380b700 1 -- 172.21.15.92:0/1862481945 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec118160 con 0x7fc5ec0fd1d0 2023-06-23T01:04:22.990 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.981+0000 7fc5f380b700 1 -- 172.21.15.92:0/1862481945 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec1032f0 con 0x7fc5ec0fd1d0 2023-06-23T01:04:22.990 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.982+0000 7fc5e1ffb700 1 -- 172.21.15.92:0/1862481945 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fc5e400fc80 con 0x7fc5ec0fd1d0 2023-06-23T01:04:22.990 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.983+0000 7fc5e1ffb700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc5d809b700 0x7fc5d809dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:22.990 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.983+0000 7fc5e1ffb700 1 -- 172.21.15.92:0/1862481945 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3694+0+0 (secure 0 0 0) 0x7fc5e40c38d0 con 0x7fc5ec0fd1d0 2023-06-23T01:04:22.991 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.983+0000 7fc5f15a7700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc5d809b700 0x7fc5d809dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:22.991 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.984+0000 7fc5f15a7700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc5d809b700 0x7fc5d809dbb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8004dc0 tx=0x7fc5e8004d10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:04:22.991 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:22.987+0000 7fc5e1ffb700 1 -- 172.21.15.92:0/1862481945 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fc5e40902a0 con 0x7fc5ec0fd1d0 2023-06-23T01:04:23.181 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:23.178+0000 7fc5f380b700 1 -- 172.21.15.92:0/1862481945 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5ec02b300 con 0x7fc5d809b700 2023-06-23T01:04:23.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:04:23.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:04:23.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:23.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:23 smithi046 ceph-mon[84949]: osdmap e27: 4 total, 4 up, 4 in 2023-06-23T01:04:23.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:04:23.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:04:23.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:23.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:23 smithi092 ceph-mon[85670]: osdmap e27: 4 total, 4 up, 4 in 2023-06-23T01:04:23.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:04:23.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:04:23.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:23.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:23 smithi131 ceph-mon[86495]: osdmap e27: 4 total, 4 up, 4 in 2023-06-23T01:04:24.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:24 smithi092 ceph-mon[85670]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:04:24.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:24 smithi092 ceph-mon[85670]: pgmap v91: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-06-23T01:04:24.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:24 smithi046 ceph-mon[84949]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:04:24.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:24 smithi046 ceph-mon[84949]: pgmap v91: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-06-23T01:04:24.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:24 smithi131 ceph-mon[86495]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:04:24.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:24 smithi131 ceph-mon[86495]: pgmap v91: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-06-23T01:04:25.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:25 smithi046 ceph-mon[84949]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-06-23T01:04:25.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:25 smithi046 ceph-mon[84949]: osdmap e28: 4 total, 4 up, 4 in 2023-06-23T01:04:25.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:25 smithi046 ceph-mon[84949]: from='client.? 172.21.15.92:0/3238123368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d8f982c-eb74-48a2-949e-d0ccd7732937"}]: dispatch 2023-06-23T01:04:25.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:25 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d8f982c-eb74-48a2-949e-d0ccd7732937"}]: dispatch 2023-06-23T01:04:25.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:25 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d8f982c-eb74-48a2-949e-d0ccd7732937"}]': finished 2023-06-23T01:04:25.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:25 smithi046 ceph-mon[84949]: osdmap e29: 5 total, 4 up, 5 in 2023-06-23T01:04:25.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:25 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:25.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:25 smithi092 ceph-mon[85670]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-06-23T01:04:25.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:25 smithi092 ceph-mon[85670]: osdmap e28: 4 total, 4 up, 4 in 2023-06-23T01:04:25.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:25 smithi092 ceph-mon[85670]: from='client.? 172.21.15.92:0/3238123368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d8f982c-eb74-48a2-949e-d0ccd7732937"}]: dispatch 2023-06-23T01:04:25.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:25 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d8f982c-eb74-48a2-949e-d0ccd7732937"}]: dispatch 2023-06-23T01:04:25.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:25 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d8f982c-eb74-48a2-949e-d0ccd7732937"}]': finished 2023-06-23T01:04:25.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:25 smithi092 ceph-mon[85670]: osdmap e29: 5 total, 4 up, 5 in 2023-06-23T01:04:25.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:25 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:25.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:25 smithi131 ceph-mon[86495]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-06-23T01:04:25.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:25 smithi131 ceph-mon[86495]: osdmap e28: 4 total, 4 up, 4 in 2023-06-23T01:04:25.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:25 smithi131 ceph-mon[86495]: from='client.? 172.21.15.92:0/3238123368' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d8f982c-eb74-48a2-949e-d0ccd7732937"}]: dispatch 2023-06-23T01:04:25.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:25 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d8f982c-eb74-48a2-949e-d0ccd7732937"}]: dispatch 2023-06-23T01:04:25.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:25 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d8f982c-eb74-48a2-949e-d0ccd7732937"}]': finished 2023-06-23T01:04:25.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:25 smithi131 ceph-mon[86495]: osdmap e29: 5 total, 4 up, 5 in 2023-06-23T01:04:25.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:25 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:26.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:26 smithi046 ceph-mon[84949]: pgmap v94: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-06-23T01:04:26.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:26 smithi046 ceph-mon[84949]: from='client.? 172.21.15.92:0/2729862752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:04:26.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:26 smithi092 ceph-mon[85670]: pgmap v94: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-06-23T01:04:26.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:26 smithi092 ceph-mon[85670]: from='client.? 172.21.15.92:0/2729862752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:04:26.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:26 smithi131 ceph-mon[86495]: pgmap v94: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-06-23T01:04:26.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:26 smithi131 ceph-mon[86495]: from='client.? 172.21.15.92:0/2729862752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:04:28.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:28 smithi131 ceph-mon[86495]: pgmap v95: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-06-23T01:04:28.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:28 smithi046 ceph-mon[84949]: pgmap v95: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-06-23T01:04:28.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:28 smithi092 ceph-mon[85670]: pgmap v95: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail; 2/6 objects degraded (33.333%) 2023-06-23T01:04:30.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:29 smithi131 ceph-mon[86495]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-06-23T01:04:30.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:29 smithi131 ceph-mon[86495]: Cluster is now healthy 2023-06-23T01:04:30.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:29 smithi092 ceph-mon[85670]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-06-23T01:04:30.234 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:29 smithi092 ceph-mon[85670]: Cluster is now healthy 2023-06-23T01:04:30.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:29 smithi046 ceph-mon[84949]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-06-23T01:04:30.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:29 smithi046 ceph-mon[84949]: Cluster is now healthy 2023-06-23T01:04:31.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:30 smithi131 ceph-mon[86495]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-06-23T01:04:31.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:30 smithi046 ceph-mon[84949]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-06-23T01:04:31.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:30 smithi092 ceph-mon[85670]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-06-23T01:04:32.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:31 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-06-23T01:04:32.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:31 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:32.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:31 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-06-23T01:04:32.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:31 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:32.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:31 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-06-23T01:04:32.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:31 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:33.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:32 smithi092 ceph-mon[85670]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:04:33.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:32 smithi092 ceph-mon[85670]: Deploying daemon osd.4 on smithi092 2023-06-23T01:04:33.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:32 smithi131 ceph-mon[86495]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:04:33.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:32 smithi131 ceph-mon[86495]: Deploying daemon osd.4 on smithi092 2023-06-23T01:04:33.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:32 smithi046 ceph-mon[84949]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:04:33.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:32 smithi046 ceph-mon[84949]: Deploying daemon osd.4 on smithi092 2023-06-23T01:04:35.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:34 smithi092 ceph-mon[85670]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-06-23T01:04:35.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:34 smithi131 ceph-mon[86495]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-06-23T01:04:35.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:34 smithi046 ceph-mon[84949]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-06-23T01:04:37.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:36 smithi131 ceph-mon[86495]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-06-23T01:04:37.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:37.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:37.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:37.289 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:36 smithi046 ceph-mon[84949]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-06-23T01:04:37.289 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:37.289 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:37.289 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:37.304 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:36 smithi092 ceph-mon[85670]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-06-23T01:04:37.304 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:37.304 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:37.304 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:38.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:38 smithi131 ceph-mon[86495]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:04:38.799 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:38 smithi092 ceph-mon[85670]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:04:38.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:38 smithi046 ceph-mon[84949]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:04:39.031 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.029+0000 7fc5e1ffb700 1 -- 172.21.15.92:0/1862481945 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc5ec02b300 con 0x7fc5d809b700 2023-06-23T01:04:39.031 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 4 on host 'smithi092' 2023-06-23T01:04:39.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 -- 172.21.15.92:0/1862481945 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc5d809b700 msgr2=0x7fc5d809dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:39.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc5d809b700 0x7fc5d809dbb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8004dc0 tx=0x7fc5e8004d10 comp rx=0 tx=0).stop 2023-06-23T01:04:39.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 -- 172.21.15.92:0/1862481945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5ec0fd1d0 msgr2=0x7fc5ec104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:39.034 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5ec0fd1d0 0x7fc5ec104c80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400b810 tx=0x7fc5e40091d0 comp rx=0 tx=0).stop 2023-06-23T01:04:39.034 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 -- 172.21.15.92:0/1862481945 shutdown_connections 2023-06-23T01:04:39.034 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fc5d809b700 0x7fc5d809dbb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:39.034 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5ec0fd1d0 0x7fc5ec104c80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:39.034 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5ec0fbe50 0x7fc5ec106090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:39.034 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 --2- 172.21.15.92:0/1862481945 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fc5ec0fc800 0x7fc5ec104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:39.034 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 -- 172.21.15.92:0/1862481945 >> 172.21.15.92:0/1862481945 conn(0x7fc5ec0f6a50 msgr2=0x7fc5ec100470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:04:39.035 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 -- 172.21.15.92:0/1862481945 shutdown_connections 2023-06-23T01:04:39.035 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:04:39.031+0000 7fc5f380b700 1 -- 172.21.15.92:0/1862481945 wait complete. 2023-06-23T01:04:39.435 DEBUG:teuthology.orchestra.run.smithi092:osd.4> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.4.service 2023-06-23T01:04:39.437 INFO:tasks.cephadm:Deploying osd.5 on smithi131 with /dev/vg_nvme/lv_3... 2023-06-23T01:04:39.437 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-06-23T01:04:39.470 INFO:journalctl@ceph.osd.4.smithi092.stdout:-- Logs begin at Fri 2023-06-23 00:44:08 UTC. -- 2023-06-23T01:04:39.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:39.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:39.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:39.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:39.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:39.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:39.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:39.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:39.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:39.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:39.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:39.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:39.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:39.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:39.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:39.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:40.030 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:04:40.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:40.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:40.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:40.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:40.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:40.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:40.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:40.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:40.582 INFO:journalctl@ceph.osd.4.smithi092.stdout:Jun 23 01:04:40 smithi092 conmon[99058]: 2023-06-23T01:04:40.171+0000 7f8021288540 -1 osd.4 0 log_to_monitors true 2023-06-23T01:04:40.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:40 smithi092 ceph-mon[85670]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:04:40.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:40 smithi092 ceph-mon[85670]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-06-23T01:04:40.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:40 smithi131 ceph-mon[86495]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:04:40.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:40 smithi131 ceph-mon[86495]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-06-23T01:04:41.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:40 smithi046 ceph-mon[84949]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:04:41.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:40 smithi046 ceph-mon[84949]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-06-23T01:04:41.488 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-06-23T01:04:41.488 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-06-23T01:04:41.488 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2023-06-23T01:04:41.489 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2023-06-23T01:04:41.489 INFO:teuthology.orchestra.run.smithi131.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113195 s, 926 MB/s 2023-06-23T01:04:41.489 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2023-06-23T01:04:41.858 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:41 smithi131 ceph-mon[86495]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-06-23T01:04:41.858 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:41 smithi131 ceph-mon[86495]: osdmap e30: 5 total, 4 up, 5 in 2023-06-23T01:04:41.858 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:41 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:41.859 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:41 smithi131 ceph-mon[86495]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:04:41.859 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:41 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:41.859 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:41 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:41.859 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:41 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:41.859 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:41 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:41.859 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:41 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:41.860 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi131:vg_nvme/lv_3 2023-06-23T01:04:41.889 INFO:journalctl@ceph.osd.4.smithi092.stdout:Jun 23 01:04:41 smithi092 conmon[99058]: 2023-06-23T01:04:41.586+0000 7f8017a6d700 -1 osd.4 0 waiting for initial osdmap 2023-06-23T01:04:41.890 INFO:journalctl@ceph.osd.4.smithi092.stdout:Jun 23 01:04:41 smithi092 conmon[99058]: 2023-06-23T01:04:41.594+0000 7f8012868700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:04:41.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:41 smithi092 ceph-mon[85670]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-06-23T01:04:41.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:41 smithi092 ceph-mon[85670]: osdmap e30: 5 total, 4 up, 5 in 2023-06-23T01:04:41.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:41 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:41.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:41 smithi092 ceph-mon[85670]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:04:41.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:41 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:41.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:41 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:41.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:41 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:41.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:41 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:41.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:41 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:42.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:41 smithi046 ceph-mon[84949]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-06-23T01:04:42.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:41 smithi046 ceph-mon[84949]: osdmap e30: 5 total, 4 up, 5 in 2023-06-23T01:04:42.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:41 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:42.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:41 smithi046 ceph-mon[84949]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:04:42.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:41 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:42.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:41 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:42.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:41 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:42.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:41 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:04:42.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:41 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:42.443 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:04:42.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:42 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi092 2023-06-23T01:04:42.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:42 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi092 to 8584M 2023-06-23T01:04:42.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:42.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:42.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:42 smithi092 ceph-mon[85670]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:04:42.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:42 smithi092 ceph-mon[85670]: osdmap e31: 5 total, 4 up, 5 in 2023-06-23T01:04:42.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:42.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:42.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:42.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:42 smithi092 ceph-mon[85670]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-06-23T01:04:42.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:42.957 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:42 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi092 2023-06-23T01:04:42.957 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:42 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi092 to 8584M 2023-06-23T01:04:42.957 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:42.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:42.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:42 smithi131 ceph-mon[86495]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:04:42.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:42 smithi131 ceph-mon[86495]: osdmap e31: 5 total, 4 up, 5 in 2023-06-23T01:04:42.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:42.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:42.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:42.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:42 smithi131 ceph-mon[86495]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-06-23T01:04:42.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:43.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:42 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi092 2023-06-23T01:04:43.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:42 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi092 to 8584M 2023-06-23T01:04:43.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:43.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:43.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:42 smithi046 ceph-mon[84949]: from='osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:04:43.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:42 smithi046 ceph-mon[84949]: osdmap e31: 5 total, 4 up, 5 in 2023-06-23T01:04:43.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:43.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:43.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:43.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:42 smithi046 ceph-mon[84949]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-06-23T01:04:43.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:43.528 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.523+0000 7f0571a8e700 1 -- 172.21.15.131:0/2688487024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f056c0fcd50 msgr2=0x7f056c0fd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:43.528 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.523+0000 7f0571a8e700 1 --2- 172.21.15.131:0/2688487024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f056c0fcd50 0x7f056c0fd1a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f05540049c0 tx=0x7f0554009e30 comp rx=0 tx=0).stop 2023-06-23T01:04:43.528 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.523+0000 7f0571a8e700 1 -- 172.21.15.131:0/2688487024 shutdown_connections 2023-06-23T01:04:43.528 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.523+0000 7f0571a8e700 1 --2- 172.21.15.131:0/2688487024 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f056c0fd6e0 0x7f056c103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:43.529 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.523+0000 7f0571a8e700 1 --2- 172.21.15.131:0/2688487024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f056c0fbb40 0x7f056c0fbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:43.529 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.523+0000 7f0571a8e700 1 --2- 172.21.15.131:0/2688487024 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f056c0fcd50 0x7f056c0fd1a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:43.529 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.523+0000 7f0571a8e700 1 -- 172.21.15.131:0/2688487024 >> 172.21.15.131:0/2688487024 conn(0x7f056c0f72c0 msgr2=0x7f056c0f96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:04:43.529 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.523+0000 7f0571a8e700 1 -- 172.21.15.131:0/2688487024 shutdown_connections 2023-06-23T01:04:43.529 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.523+0000 7f0571a8e700 1 -- 172.21.15.131:0/2688487024 wait complete. 2023-06-23T01:04:43.529 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f0571a8e700 1 Processor -- start 2023-06-23T01:04:43.529 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f0571a8e700 1 -- start start 2023-06-23T01:04:43.529 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f0571a8e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f056c0fbb40 0x7f056c10d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:43.530 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f0571a8e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f056c0fcd50 0x7f056c10dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:43.530 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f0571a8e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f056c0fd6e0 0x7f056c114d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:43.530 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f0571a8e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c1153a0 con 0x7f056c0fd6e0 2023-06-23T01:04:43.530 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f0571a8e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c115510 con 0x7f056c0fcd50 2023-06-23T01:04:43.530 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f0571a8e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c1156d0 con 0x7f056c0fbb40 2023-06-23T01:04:43.530 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f056b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f056c0fbb40 0x7f056c10d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:43.530 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f056b7fe700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f056c0fbb40 0x7f056c10d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:53614/0 (socket says 172.21.15.131:53614) 2023-06-23T01:04:43.531 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f056b7fe700 1 -- 172.21.15.131:0/972926264 learned_addr learned my addr 172.21.15.131:0/972926264 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-06-23T01:04:43.531 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f0562ffd700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f056c0fcd50 0x7f056c10dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:43.531 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f056bfff700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f056c0fd6e0 0x7f056c114d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:43.531 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f056b7fe700 1 -- 172.21.15.131:0/972926264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f056c0fcd50 msgr2=0x7f056c10dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:43.531 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f056b7fe700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f056c0fcd50 0x7f056c10dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:43.531 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f056b7fe700 1 -- 172.21.15.131:0/972926264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f056c0fd6e0 msgr2=0x7f056c114d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:43.531 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f056b7fe700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f056c0fd6e0 0x7f056c114d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:43.531 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f056b7fe700 1 -- 172.21.15.131:0/972926264 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0554005040 con 0x7f056c0fbb40 2023-06-23T01:04:43.532 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.524+0000 7f0562ffd700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f056c0fcd50 0x7f056c10dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:04:43.532 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.525+0000 7f056b7fe700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f056c0fbb40 0x7f056c10d7c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f055c00ebf0 tx=0x7f055c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:04:43.532 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.525+0000 7f0568ff9700 1 -- 172.21.15.131:0/972926264 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c007cc0 con 0x7f056c0fbb40 2023-06-23T01:04:43.532 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.525+0000 7f0571a8e700 1 -- 172.21.15.131:0/972926264 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056c115950 con 0x7f056c0fbb40 2023-06-23T01:04:43.532 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.525+0000 7f0568ff9700 1 -- 172.21.15.131:0/972926264 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f055c00f040 con 0x7f056c0fbb40 2023-06-23T01:04:43.532 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.525+0000 7f0568ff9700 1 -- 172.21.15.131:0/972926264 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055c00c800 con 0x7f056c0fbb40 2023-06-23T01:04:43.532 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.525+0000 7f0571a8e700 1 -- 172.21.15.131:0/972926264 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056c115ec0 con 0x7f056c0fbb40 2023-06-23T01:04:43.533 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.526+0000 7f0571a8e700 1 -- 172.21.15.131:0/972926264 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056c1071e0 con 0x7f056c0fbb40 2023-06-23T01:04:43.533 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.527+0000 7f0568ff9700 1 -- 172.21.15.131:0/972926264 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f055c00cc30 con 0x7f056c0fbb40 2023-06-23T01:04:43.534 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.527+0000 7f0568ff9700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f055809b680 0x7f055809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:04:43.534 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.527+0000 7f0568ff9700 1 -- 172.21.15.131:0/972926264 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4097+0+0 (secure 0 0 0) 0x7f055c0c3820 con 0x7f056c0fbb40 2023-06-23T01:04:43.534 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.527+0000 7f0562ffd700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f055809b680 0x7f055809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:04:43.534 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.528+0000 7f0562ffd700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f055809b680 0x7f055809db30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0554009b80 tx=0x7f0554009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:04:43.534 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.531+0000 7f0568ff9700 1 -- 172.21.15.131:0/972926264 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f055c04da00 con 0x7f056c0fbb40 2023-06-23T01:04:43.726 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:43.723+0000 7f0571a8e700 1 -- 172.21.15.131:0/972926264 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f056c0f8250 con 0x7f055809b680 2023-06-23T01:04:43.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:43 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:04:43.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:43 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:04:43.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:43 smithi092 ceph-mon[85670]: osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902] boot 2023-06-23T01:04:43.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:43 smithi092 ceph-mon[85670]: osdmap e32: 5 total, 5 up, 5 in 2023-06-23T01:04:43.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:43 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:43.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:43 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:04:43.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:43 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:04:43.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:43 smithi131 ceph-mon[86495]: osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902] boot 2023-06-23T01:04:43.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:43 smithi131 ceph-mon[86495]: osdmap e32: 5 total, 5 up, 5 in 2023-06-23T01:04:43.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:44.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:43 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:04:44.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:43 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:04:44.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:43 smithi046 ceph-mon[84949]: osd.4 [v2:172.21.15.92:6808/2024629902,v1:172.21.15.92:6809/2024629902] boot 2023-06-23T01:04:44.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:43 smithi046 ceph-mon[84949]: osdmap e32: 5 total, 5 up, 5 in 2023-06-23T01:04:44.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-06-23T01:04:44.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:44 smithi046 ceph-mon[84949]: osdmap e33: 5 total, 5 up, 5 in 2023-06-23T01:04:44.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:44 smithi046 ceph-mon[84949]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:04:44.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:04:44.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:04:44.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:44.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:44 smithi046 ceph-mon[84949]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:44.857 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:44 smithi131 ceph-mon[86495]: osdmap e33: 5 total, 5 up, 5 in 2023-06-23T01:04:44.857 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:44 smithi131 ceph-mon[86495]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:04:44.857 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:04:44.857 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:04:44.857 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:44.857 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:44 smithi131 ceph-mon[86495]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:44.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:44 smithi092 ceph-mon[85670]: osdmap e33: 5 total, 5 up, 5 in 2023-06-23T01:04:44.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:44 smithi092 ceph-mon[85670]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:04:44.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:04:44.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:04:44.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:44.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:44 smithi092 ceph-mon[85670]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:47.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:46 smithi092 ceph-mon[85670]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:47.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:46 smithi092 ceph-mon[85670]: from='client.? 172.21.15.131:0/1628501375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52a7160c-4735-4a77-bc08-4d411c282b37"}]: dispatch 2023-06-23T01:04:47.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:46 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52a7160c-4735-4a77-bc08-4d411c282b37"}]: dispatch 2023-06-23T01:04:47.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:46 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52a7160c-4735-4a77-bc08-4d411c282b37"}]': finished 2023-06-23T01:04:47.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:46 smithi092 ceph-mon[85670]: osdmap e34: 6 total, 5 up, 6 in 2023-06-23T01:04:47.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:04:47.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:46 smithi092 ceph-mon[85670]: from='client.? 172.21.15.131:0/1382501125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:04:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:46 smithi131 ceph-mon[86495]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:46 smithi131 ceph-mon[86495]: from='client.? 172.21.15.131:0/1628501375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52a7160c-4735-4a77-bc08-4d411c282b37"}]: dispatch 2023-06-23T01:04:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:46 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52a7160c-4735-4a77-bc08-4d411c282b37"}]: dispatch 2023-06-23T01:04:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:46 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52a7160c-4735-4a77-bc08-4d411c282b37"}]': finished 2023-06-23T01:04:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:46 smithi131 ceph-mon[86495]: osdmap e34: 6 total, 5 up, 6 in 2023-06-23T01:04:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:04:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:46 smithi131 ceph-mon[86495]: from='client.? 172.21.15.131:0/1382501125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:04:47.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:46 smithi046 ceph-mon[84949]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:47.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:46 smithi046 ceph-mon[84949]: from='client.? 172.21.15.131:0/1628501375' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52a7160c-4735-4a77-bc08-4d411c282b37"}]: dispatch 2023-06-23T01:04:47.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:46 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52a7160c-4735-4a77-bc08-4d411c282b37"}]: dispatch 2023-06-23T01:04:47.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:46 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52a7160c-4735-4a77-bc08-4d411c282b37"}]': finished 2023-06-23T01:04:47.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:46 smithi046 ceph-mon[84949]: osdmap e34: 6 total, 5 up, 6 in 2023-06-23T01:04:47.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:04:47.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:46 smithi046 ceph-mon[84949]: from='client.? 172.21.15.131:0/1382501125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:04:48.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:48 smithi131 ceph-mon[86495]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:48.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:48 smithi046 ceph-mon[84949]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:48.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:48 smithi092 ceph-mon[85670]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:51.122 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:50 smithi131 ceph-mon[86495]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:51.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:50 smithi092 ceph-mon[85670]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:51.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:50 smithi046 ceph-mon[84949]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:53.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:52 smithi092 ceph-mon[85670]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:53.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:52 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-06-23T01:04:53.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:52 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:53.164 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:52 smithi131 ceph-mon[86495]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:53.164 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:52 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-06-23T01:04:53.164 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:52 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:53.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:52 smithi046 ceph-mon[84949]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:53.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:52 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-06-23T01:04:53.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:52 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:54.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:53 smithi092 ceph-mon[85670]: Deploying daemon osd.5 on smithi131 2023-06-23T01:04:54.150 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:53 smithi131 ceph-mon[86495]: Deploying daemon osd.5 on smithi131 2023-06-23T01:04:54.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:53 smithi046 ceph-mon[84949]: Deploying daemon osd.5 on smithi131 2023-06-23T01:04:55.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:54 smithi092 ceph-mon[85670]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:55.187 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:54 smithi131 ceph-mon[86495]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:55.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:54 smithi046 ceph-mon[84949]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:57.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:56 smithi092 ceph-mon[85670]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:57.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:57.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:57.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:56 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:57.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:56 smithi131 ceph-mon[86495]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:57.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:57.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:57.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:56 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:57.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:56 smithi046 ceph-mon[84949]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:57.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:57.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:57.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:56 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:58.523 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:58 smithi131 ceph-mon[86495]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:58.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:58 smithi046 ceph-mon[84949]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:58.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:58 smithi092 ceph-mon[85670]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:04:59.336 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 5 on host 'smithi131' 2023-06-23T01:04:59.337 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.333+0000 7f0568ff9700 1 -- 172.21.15.131:0/972926264 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f056c0f8250 con 0x7f055809b680 2023-06-23T01:04:59.338 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 -- 172.21.15.131:0/972926264 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f055809b680 msgr2=0x7f055809db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:59.338 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f055809b680 0x7f055809db30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0554009b80 tx=0x7f0554009e30 comp rx=0 tx=0).stop 2023-06-23T01:04:59.339 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 -- 172.21.15.131:0/972926264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f056c0fbb40 msgr2=0x7f056c10d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:04:59.339 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f056c0fbb40 0x7f056c10d7c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f055c00ebf0 tx=0x7f055c00b040 comp rx=0 tx=0).stop 2023-06-23T01:04:59.339 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 -- 172.21.15.131:0/972926264 shutdown_connections 2023-06-23T01:04:59.339 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f056c0fd6e0 0x7f056c114d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:59.339 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f056c0fbb40 0x7f056c10d7c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:59.339 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f055809b680 0x7f055809db30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:59.339 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 --2- 172.21.15.131:0/972926264 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f056c0fcd50 0x7f056c10dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:04:59.340 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 -- 172.21.15.131:0/972926264 >> 172.21.15.131:0/972926264 conn(0x7f056c0f72c0 msgr2=0x7f056c0f7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:04:59.340 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 -- 172.21.15.131:0/972926264 shutdown_connections 2023-06-23T01:04:59.340 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:04:59.335+0000 7f0571a8e700 1 -- 172.21.15.131:0/972926264 wait complete. 2023-06-23T01:04:59.706 DEBUG:teuthology.orchestra.run.smithi131:osd.5> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.5.service 2023-06-23T01:04:59.708 INFO:tasks.cephadm:Deploying osd.6 on smithi046 with /dev/vg_nvme/lv_2... 2023-06-23T01:04:59.709 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-06-23T01:04:59.749 INFO:journalctl@ceph.osd.5.smithi131.stdout:-- Logs begin at Fri 2023-06-23 00:44:02 UTC. -- 2023-06-23T01:04:59.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:59.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:59.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:04:59.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:04:59.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:59.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:04:59.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:04:59.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:04:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:00.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:00.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:00.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:00.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:00.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:00.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:00.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:00.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:04:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:00.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:00.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:00.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:00.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:00.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:00.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:00.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:00.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:04:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:00.333 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:05:00.957 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:00 smithi131 ceph-mon[86495]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:05:00.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:00 smithi131 ceph-mon[86495]: from='osd.5 [v2:172.21.15.131:6808/2577961971,v1:172.21.15.131:6809/2577961971]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-23T01:05:00.958 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:00 smithi131 ceph-mon[86495]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-23T01:05:00.958 INFO:journalctl@ceph.osd.5.smithi131.stdout:Jun 23 01:05:00 smithi131 conmon[101153]: 2023-06-23T01:05:00.604+0000 7f1be6968540 -1 osd.5 0 log_to_monitors true 2023-06-23T01:05:01.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:00 smithi046 ceph-mon[84949]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:05:01.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:00 smithi046 ceph-mon[84949]: from='osd.5 [v2:172.21.15.131:6808/2577961971,v1:172.21.15.131:6809/2577961971]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-23T01:05:01.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:00 smithi046 ceph-mon[84949]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-23T01:05:01.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:00 smithi092 ceph-mon[85670]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:05:01.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:00 smithi092 ceph-mon[85670]: from='osd.5 [v2:172.21.15.131:6808/2577961971,v1:172.21.15.131:6809/2577961971]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-23T01:05:01.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:00 smithi092 ceph-mon[85670]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-06-23T01:05:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-06-23T01:05:01.752 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-06-23T01:05:01.753 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2023-06-23T01:05:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2023-06-23T01:05:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00857855 s, 1.2 GB/s 2023-06-23T01:05:01.753 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2023-06-23T01:05:02.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:01 smithi046 ceph-mon[84949]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-06-23T01:05:02.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:01 smithi046 ceph-mon[84949]: osdmap e35: 6 total, 5 up, 6 in 2023-06-23T01:05:02.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:01 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:02.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:01 smithi046 ceph-mon[84949]: from='osd.5 [v2:172.21.15.131:6808/2577961971,v1:172.21.15.131:6809/2577961971]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:05:02.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:01 smithi046 ceph-mon[84949]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:05:02.144 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:01 smithi092 ceph-mon[85670]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-06-23T01:05:02.145 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:01 smithi092 ceph-mon[85670]: osdmap e35: 6 total, 5 up, 6 in 2023-06-23T01:05:02.145 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:01 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:02.145 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:01 smithi092 ceph-mon[85670]: from='osd.5 [v2:172.21.15.131:6808/2577961971,v1:172.21.15.131:6809/2577961971]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:05:02.145 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:01 smithi092 ceph-mon[85670]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:05:02.172 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi046:vg_nvme/lv_2 2023-06-23T01:05:02.236 INFO:journalctl@ceph.osd.5.smithi131.stdout:Jun 23 01:05:01 smithi131 conmon[101153]: 2023-06-23T01:05:01.776+0000 7f1bdd14d700 -1 osd.5 0 waiting for initial osdmap 2023-06-23T01:05:02.237 INFO:journalctl@ceph.osd.5.smithi131.stdout:Jun 23 01:05:01 smithi131 conmon[101153]: 2023-06-23T01:05:01.783+0000 7f1bd8f4a700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:05:02.237 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:01 smithi131 ceph-mon[86495]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-06-23T01:05:02.237 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:01 smithi131 ceph-mon[86495]: osdmap e35: 6 total, 5 up, 6 in 2023-06-23T01:05:02.237 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:01 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:02.238 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:01 smithi131 ceph-mon[86495]: from='osd.5 [v2:172.21.15.131:6808/2577961971,v1:172.21.15.131:6809/2577961971]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:05:02.238 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:01 smithi131 ceph-mon[86495]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:05:02.766 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:05:03.052 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:05:03.052 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: osdmap e36: 6 total, 5 up, 6 in 2023-06-23T01:05:03.052 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:03.052 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:03.052 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:05:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi131 2023-06-23T01:05:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi131 to 8584M 2023-06-23T01:05:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:03.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:03.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: osd.5 [v2:172.21.15.131:6808/2577961971,v1:172.21.15.131:6809/2577961971] boot 2023-06-23T01:05:03.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: osdmap e37: 6 total, 6 up, 6 in 2023-06-23T01:05:03.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:03.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:05:03.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: osdmap e36: 6 total, 5 up, 6 in 2023-06-23T01:05:03.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:03.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:03.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:05:03.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi131 2023-06-23T01:05:03.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:03.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:03.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi131 to 8584M 2023-06-23T01:05:03.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:03.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:03.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: osd.5 [v2:172.21.15.131:6808/2577961971,v1:172.21.15.131:6809/2577961971] boot 2023-06-23T01:05:03.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: osdmap e37: 6 total, 6 up, 6 in 2023-06-23T01:05:03.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:03.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:05:03.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: osdmap e36: 6 total, 5 up, 6 in 2023-06-23T01:05:03.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:03.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:03.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-06-23T01:05:03.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi131 2023-06-23T01:05:03.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:03.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:03.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi131 to 8584M 2023-06-23T01:05:03.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:03.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:03.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:03.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: osd.5 [v2:172.21.15.131:6808/2577961971,v1:172.21.15.131:6809/2577961971] boot 2023-06-23T01:05:03.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: osdmap e37: 6 total, 6 up, 6 in 2023-06-23T01:05:03.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-06-23T01:05:03.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.979+0000 7f035075b700 1 -- 172.21.15.46:0/1728135477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03480fbb40 msgr2=0x7f03480fbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:03.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.979+0000 7f035075b700 1 --2- 172.21.15.46:0/1728135477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03480fbb40 0x7f03480fbf50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0338009d60 tx=0x7f0338004d10 comp rx=0 tx=0).stop 2023-06-23T01:05:03.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.979+0000 7f035075b700 1 -- 172.21.15.46:0/1728135477 shutdown_connections 2023-06-23T01:05:03.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.979+0000 7f035075b700 1 --2- 172.21.15.46:0/1728135477 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03480fd7a0 0x7f0348104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:03.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.979+0000 7f035075b700 1 --2- 172.21.15.46:0/1728135477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03480fbb40 0x7f03480fbf50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:03.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.979+0000 7f035075b700 1 --2- 172.21.15.46:0/1728135477 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03480fcdf0 0x7f03480fd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:03.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.979+0000 7f035075b700 1 -- 172.21.15.46:0/1728135477 >> 172.21.15.46:0/1728135477 conn(0x7f03480f72a0 msgr2=0x7f03480f96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:05:03.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.979+0000 7f035075b700 1 -- 172.21.15.46:0/1728135477 shutdown_connections 2023-06-23T01:05:03.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.979+0000 7f035075b700 1 -- 172.21.15.46:0/1728135477 wait complete. 2023-06-23T01:05:03.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f035075b700 1 Processor -- start 2023-06-23T01:05:03.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f035075b700 1 -- start start 2023-06-23T01:05:03.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f035075b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03480fbb40 0x7f034810d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:03.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f035075b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03480fcdf0 0x7f034810ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:03.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f035075b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03480fd7a0 0x7f0348114df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:03.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f035075b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0348115400 con 0x7f03480fd7a0 2023-06-23T01:05:03.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f035075b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0348115570 con 0x7f03480fbb40 2023-06-23T01:05:03.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f035075b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0348115730 con 0x7f03480fcdf0 2023-06-23T01:05:03.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034ecf8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03480fd7a0 0x7f0348114df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:03.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034ecf8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03480fd7a0 0x7f0348114df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58642/0 (socket says 172.21.15.46:58642) 2023-06-23T01:05:03.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034ecf8700 1 -- 172.21.15.46:0/1301407694 learned_addr learned my addr 172.21.15.46:0/1301407694 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:05:03.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034e4f7700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03480fbb40 0x7f034810d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:03.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034dcf6700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03480fcdf0 0x7f034810ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:03.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034ecf8700 1 -- 172.21.15.46:0/1301407694 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03480fcdf0 msgr2=0x7f034810ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:03.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034ecf8700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03480fcdf0 0x7f034810ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:03.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034ecf8700 1 -- 172.21.15.46:0/1301407694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03480fbb40 msgr2=0x7f034810d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:03.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034ecf8700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03480fbb40 0x7f034810d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:03.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034ecf8700 1 -- 172.21.15.46:0/1301407694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0338009940 con 0x7f03480fd7a0 2023-06-23T01:05:03.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034dcf6700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03480fcdf0 0x7f034810ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:05:03.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.980+0000 7f034ecf8700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03480fd7a0 0x7f0348114df0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0340002b00 tx=0x7f03400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:05:03.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.981+0000 7f0346ffd700 1 -- 172.21.15.46:0/1301407694 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340007cc0 con 0x7f03480fd7a0 2023-06-23T01:05:03.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.981+0000 7f0346ffd700 1 -- 172.21.15.46:0/1301407694 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f034000f040 con 0x7f03480fd7a0 2023-06-23T01:05:03.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.981+0000 7f0346ffd700 1 -- 172.21.15.46:0/1301407694 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034000d830 con 0x7f03480fd7a0 2023-06-23T01:05:03.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.981+0000 7f035075b700 1 -- 172.21.15.46:0/1301407694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0348104ae0 con 0x7f03480fd7a0 2023-06-23T01:05:03.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.981+0000 7f035075b700 1 -- 172.21.15.46:0/1301407694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0348104f80 con 0x7f03480fd7a0 2023-06-23T01:05:03.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.982+0000 7f035075b700 1 -- 172.21.15.46:0/1301407694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0348103200 con 0x7f03480fd7a0 2023-06-23T01:05:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.983+0000 7f0346ffd700 1 -- 172.21.15.46:0/1301407694 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f034000dc60 con 0x7f03480fd7a0 2023-06-23T01:05:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.984+0000 7f0346ffd700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f033009b700 0x7f033009dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.984+0000 7f034e4f7700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f033009b700 0x7f033009dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.984+0000 7f0346ffd700 1 -- 172.21.15.46:0/1301407694 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4558+0+0 (secure 0 0 0) 0x7f03400c3c50 con 0x7f03480fd7a0 2023-06-23T01:05:03.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.984+0000 7f034e4f7700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f033009b700 0x7f033009dbb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0338004dc0 tx=0x7f0338004d10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:05:03.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:03.988+0000 7f0346ffd700 1 -- 172.21.15.46:0/1301407694 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f03400c9020 con 0x7f03480fd7a0 2023-06-23T01:05:04.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:04.191+0000 7f035075b700 1 -- 172.21.15.46:0/1301407694 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f034802b1f0 con 0x7f033009b700 2023-06-23T01:05:04.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:04 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:05:04.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:04 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:05:04.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:04 smithi131 ceph-mon[86495]: osdmap e38: 6 total, 6 up, 6 in 2023-06-23T01:05:04.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:04 smithi131 ceph-mon[86495]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:04.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:05:04.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:05:04.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:04.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:04 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:05:04.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:04 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:05:04.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:04 smithi046 ceph-mon[84949]: osdmap e38: 6 total, 6 up, 6 in 2023-06-23T01:05:04.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:04 smithi046 ceph-mon[84949]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:04.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:05:04.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:05:04.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:04.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:04 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:05:04.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:04 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:05:04.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:04 smithi092 ceph-mon[85670]: osdmap e38: 6 total, 6 up, 6 in 2023-06-23T01:05:04.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:04 smithi092 ceph-mon[85670]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:04.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:05:04.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:05:04.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:05.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:05 smithi131 ceph-mon[86495]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:05:05.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:05 smithi131 ceph-mon[86495]: osdmap e39: 6 total, 6 up, 6 in 2023-06-23T01:05:05.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:05 smithi046 ceph-mon[84949]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:05:05.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:05 smithi046 ceph-mon[84949]: osdmap e39: 6 total, 6 up, 6 in 2023-06-23T01:05:05.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:05 smithi092 ceph-mon[85670]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:05:05.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:05 smithi092 ceph-mon[85670]: osdmap e39: 6 total, 6 up, 6 in 2023-06-23T01:05:06.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:06 smithi131 ceph-mon[86495]: osdmap e40: 6 total, 6 up, 6 in 2023-06-23T01:05:06.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:06 smithi131 ceph-mon[86495]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:06.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:06 smithi046 ceph-mon[84949]: osdmap e40: 6 total, 6 up, 6 in 2023-06-23T01:05:06.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:06 smithi046 ceph-mon[84949]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:06.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:06 smithi092 ceph-mon[85670]: osdmap e40: 6 total, 6 up, 6 in 2023-06-23T01:05:06.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:06 smithi092 ceph-mon[85670]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:07.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:07 smithi131 ceph-mon[86495]: osdmap e41: 6 total, 6 up, 6 in 2023-06-23T01:05:07.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:07 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/3255086030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f3d2690-f655-4dda-a36c-bc97f9d08a27"}]: dispatch 2023-06-23T01:05:07.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:07 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/3255086030' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f3d2690-f655-4dda-a36c-bc97f9d08a27"}]': finished 2023-06-23T01:05:07.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:07 smithi131 ceph-mon[86495]: osdmap e42: 7 total, 6 up, 7 in 2023-06-23T01:05:07.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:07.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:07 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/1243136267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:05:07.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:07 smithi046 ceph-mon[84949]: osdmap e41: 6 total, 6 up, 6 in 2023-06-23T01:05:07.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:07 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3255086030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f3d2690-f655-4dda-a36c-bc97f9d08a27"}]: dispatch 2023-06-23T01:05:07.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:07 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3255086030' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f3d2690-f655-4dda-a36c-bc97f9d08a27"}]': finished 2023-06-23T01:05:07.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:07 smithi046 ceph-mon[84949]: osdmap e42: 7 total, 6 up, 7 in 2023-06-23T01:05:07.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:07.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:07 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/1243136267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:05:07.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:07 smithi092 ceph-mon[85670]: osdmap e41: 6 total, 6 up, 6 in 2023-06-23T01:05:07.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:07 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/3255086030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f3d2690-f655-4dda-a36c-bc97f9d08a27"}]: dispatch 2023-06-23T01:05:07.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:07 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/3255086030' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f3d2690-f655-4dda-a36c-bc97f9d08a27"}]': finished 2023-06-23T01:05:07.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:07 smithi092 ceph-mon[85670]: osdmap e42: 7 total, 6 up, 7 in 2023-06-23T01:05:07.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:07.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:07 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/1243136267' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:05:08.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:08 smithi131 ceph-mon[86495]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:08.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:08 smithi046 ceph-mon[84949]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:08.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:08 smithi092 ceph-mon[85670]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:11.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:10 smithi092 ceph-mon[85670]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-06-23T01:05:11.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:10 smithi131 ceph-mon[86495]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-06-23T01:05:11.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:10 smithi046 ceph-mon[84949]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-06-23T01:05:13.034 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:12 smithi046 ceph-mon[84949]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-06-23T01:05:13.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:12 smithi092 ceph-mon[85670]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-06-23T01:05:13.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:12 smithi131 ceph-mon[86495]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-06-23T01:05:14.046 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:13 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-06-23T01:05:14.046 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:13 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:14.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:13 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-06-23T01:05:14.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:13 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:14.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:13 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-06-23T01:05:14.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:13 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:15.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:14 smithi092 ceph-mon[85670]: Deploying daemon osd.6 on smithi046 2023-06-23T01:05:15.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:14 smithi092 ceph-mon[85670]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:05:15.179 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:14 smithi046 ceph-mon[84949]: Deploying daemon osd.6 on smithi046 2023-06-23T01:05:15.180 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:14 smithi046 ceph-mon[84949]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:05:15.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:14 smithi131 ceph-mon[86495]: Deploying daemon osd.6 on smithi046 2023-06-23T01:05:15.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:14 smithi131 ceph-mon[86495]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:05:17.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:16 smithi092 ceph-mon[85670]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-06-23T01:05:17.205 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:16 smithi046 ceph-mon[84949]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-06-23T01:05:17.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:16 smithi131 ceph-mon[86495]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-06-23T01:05:18.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:18.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:18.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:18 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:18.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:18 smithi046 ceph-mon[84949]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-06-23T01:05:18.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:18.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:18.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:18 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:18.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:18 smithi092 ceph-mon[85670]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-06-23T01:05:18.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:18.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:18.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:18 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:18.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:18 smithi131 ceph-mon[86495]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-06-23T01:05:20.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.289+0000 7f0346ffd700 1 -- 172.21.15.46:0/1301407694 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f034802b1f0 con 0x7f033009b700 2023-06-23T01:05:20.293 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 6 on host 'smithi046' 2023-06-23T01:05:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.291+0000 7f035075b700 1 -- 172.21.15.46:0/1301407694 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f033009b700 msgr2=0x7f033009dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.291+0000 7f035075b700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f033009b700 0x7f033009dbb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0338004dc0 tx=0x7f0338004d10 comp rx=0 tx=0).stop 2023-06-23T01:05:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.291+0000 7f035075b700 1 -- 172.21.15.46:0/1301407694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03480fd7a0 msgr2=0x7f0348114df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.291+0000 7f035075b700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03480fd7a0 0x7f0348114df0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0340002b00 tx=0x7f03400046b0 comp rx=0 tx=0).stop 2023-06-23T01:05:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.291+0000 7f035075b700 1 -- 172.21.15.46:0/1301407694 shutdown_connections 2023-06-23T01:05:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.291+0000 7f035075b700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f033009b700 0x7f033009dbb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.291+0000 7f035075b700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03480fbb40 0x7f034810d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.291+0000 7f035075b700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03480fd7a0 0x7f0348114df0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.291+0000 7f035075b700 1 --2- 172.21.15.46:0/1301407694 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03480fcdf0 0x7f034810ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.292+0000 7f035075b700 1 -- 172.21.15.46:0/1301407694 >> 172.21.15.46:0/1301407694 conn(0x7f03480f72a0 msgr2=0x7f0348100380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:05:20.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.292+0000 7f035075b700 1 -- 172.21.15.46:0/1301407694 shutdown_connections 2023-06-23T01:05:20.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:05:20.292+0000 7f035075b700 1 -- 172.21.15.46:0/1301407694 wait complete. 2023-06-23T01:05:20.725 DEBUG:teuthology.orchestra.run.smithi046:osd.6> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.6.service 2023-06-23T01:05:20.728 INFO:tasks.cephadm:Deploying osd.7 on smithi092 with /dev/vg_nvme/lv_2... 2023-06-23T01:05:20.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-06-23T01:05:20.755 INFO:journalctl@ceph.osd.6.smithi046.stdout:-- Logs begin at Fri 2023-06-23 00:44:15 UTC. -- 2023-06-23T01:05:21.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:20 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:20 smithi046 ceph-mon[84949]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:05:21.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:20 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:20 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:20 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:20 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:20 smithi092 ceph-mon[85670]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:05:21.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:20 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:20 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:20 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:20 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:20 smithi131 ceph-mon[86495]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:05:21.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:20 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:20 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:20 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:21.333 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-23T01:05:21.568 INFO:journalctl@ceph.osd.6.smithi046.stdout:Jun 23 01:05:21 smithi046 conmon[116055]: 2023-06-23T01:05:21.329+0000 7f18b2a28540 -1 osd.6 0 log_to_monitors true 2023-06-23T01:05:22.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:21 smithi046 ceph-mon[84949]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-06-23T01:05:22.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:21 smithi092 ceph-mon[85670]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-06-23T01:05:22.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:21 smithi131 ceph-mon[86495]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-06-23T01:05:22.823 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-06-23T01:05:22.823 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-06-23T01:05:22.824 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2023-06-23T01:05:22.824 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2023-06-23T01:05:22.824 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0129428 s, 810 MB/s 2023-06-23T01:05:22.824 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2023-06-23T01:05:22.945 INFO:journalctl@ceph.osd.6.smithi046.stdout:Jun 23 01:05:22 smithi046 conmon[116055]: 2023-06-23T01:05:22.839+0000 7f18a920d700 -1 osd.6 0 waiting for initial osdmap 2023-06-23T01:05:22.946 INFO:journalctl@ceph.osd.6.smithi046.stdout:Jun 23 01:05:22 smithi046 conmon[116055]: 2023-06-23T01:05:22.849+0000 7f18a4008700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:05:22.946 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:22 smithi046 ceph-mon[84949]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:22.946 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:22 smithi046 ceph-mon[84949]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-06-23T01:05:22.947 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:22 smithi046 ceph-mon[84949]: osdmap e43: 7 total, 6 up, 7 in 2023-06-23T01:05:22.947 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:22 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:22.947 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:22 smithi046 ceph-mon[84949]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:05:23.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:22 smithi092 ceph-mon[85670]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:23.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:22 smithi092 ceph-mon[85670]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-06-23T01:05:23.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:22 smithi092 ceph-mon[85670]: osdmap e43: 7 total, 6 up, 7 in 2023-06-23T01:05:23.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:22 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:23.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:22 smithi092 ceph-mon[85670]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:05:23.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:22 smithi131 ceph-mon[86495]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:23.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:22 smithi131 ceph-mon[86495]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-06-23T01:05:23.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:22 smithi131 ceph-mon[86495]: osdmap e43: 7 total, 6 up, 7 in 2023-06-23T01:05:23.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:22 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:23.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:22 smithi131 ceph-mon[86495]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:05:23.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi092:vg_nvme/lv_2 2023-06-23T01:05:23.856 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-23T01:05:24.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:05:24.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: osdmap e44: 7 total, 6 up, 7 in 2023-06-23T01:05:24.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:24.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:24.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:24.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:24.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:24.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:24.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:24.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:24.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:24.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:24.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:24.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:05:24.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: osdmap e44: 7 total, 6 up, 7 in 2023-06-23T01:05:24.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:24.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:24.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:24.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:24.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:24.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:24.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:24.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:24.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:24.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:24.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:24.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:05:24.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: osdmap e44: 7 total, 6 up, 7 in 2023-06-23T01:05:24.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:24.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:24.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:24.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:24.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:24.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:24.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:24.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:24.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:24.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:24.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:24.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:25.069 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.064+0000 7f4e5de4a700 1 -- 172.21.15.92:0/645129874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e580fbe20 msgr2=0x7f4e580fc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:25.069 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.064+0000 7f4e5de4a700 1 --2- 172.21.15.92:0/645129874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e580fbe20 0x7f4e580fc270 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c0049f0 tx=0x7f4e4c009e30 comp rx=0 tx=0).stop 2023-06-23T01:05:25.069 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.064+0000 7f4e5de4a700 1 -- 172.21.15.92:0/645129874 shutdown_connections 2023-06-23T01:05:25.069 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.064+0000 7f4e5de4a700 1 --2- 172.21.15.92:0/645129874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e580fc7b0 0x7f4e58104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:25.070 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.064+0000 7f4e5de4a700 1 --2- 172.21.15.92:0/645129874 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e580fd0d0 0x7f4e580fd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:25.070 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.064+0000 7f4e5de4a700 1 --2- 172.21.15.92:0/645129874 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e580fbe20 0x7f4e580fc270 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:25.070 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.064+0000 7f4e5de4a700 1 -- 172.21.15.92:0/645129874 >> 172.21.15.92:0/645129874 conn(0x7f4e580f6a00 msgr2=0x7f4e580f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:05:25.070 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.064+0000 7f4e5de4a700 1 -- 172.21.15.92:0/645129874 shutdown_connections 2023-06-23T01:05:25.070 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.064+0000 7f4e5de4a700 1 -- 172.21.15.92:0/645129874 wait complete. 2023-06-23T01:05:25.070 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.064+0000 7f4e5de4a700 1 Processor -- start 2023-06-23T01:05:25.070 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e5de4a700 1 -- start start 2023-06-23T01:05:25.071 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e5de4a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e580fbe20 0x7f4e5810c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:25.071 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e5de4a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e580fc7b0 0x7f4e5810cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:25.071 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e5de4a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e580fd0d0 0x7f4e58113cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:25.071 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e5de4a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e581142c0 con 0x7f4e580fd0d0 2023-06-23T01:05:25.071 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e5de4a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e58114430 con 0x7f4e580fc7b0 2023-06-23T01:05:25.071 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e5de4a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e581145f0 con 0x7f4e580fbe20 2023-06-23T01:05:25.071 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e56ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e580fc7b0 0x7f4e5810cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:25.072 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e56ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e580fc7b0 0x7f4e5810cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:58300/0 (socket says 172.21.15.92:58300) 2023-06-23T01:05:25.072 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e56ffd700 1 -- 172.21.15.92:0/1696942763 learned_addr learned my addr 172.21.15.92:0/1696942763 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:05:25.072 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e57fff700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e580fd0d0 0x7f4e58113cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:25.072 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e577fe700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e580fbe20 0x7f4e5810c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:25.072 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e56ffd700 1 -- 172.21.15.92:0/1696942763 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e580fbe20 msgr2=0x7f4e5810c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:25.072 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e56ffd700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e580fbe20 0x7f4e5810c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:25.072 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e56ffd700 1 -- 172.21.15.92:0/1696942763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e580fd0d0 msgr2=0x7f4e58113cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:25.073 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e56ffd700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e580fd0d0 0x7f4e58113cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:25.073 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e56ffd700 1 -- 172.21.15.92:0/1696942763 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c005040 con 0x7f4e580fc7b0 2023-06-23T01:05:25.073 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e577fe700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e580fbe20 0x7f4e5810c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:05:25.073 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e56ffd700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e580fc7b0 0x7f4e5810cc20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009f00 tx=0x7f4e4c009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:05:25.073 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.065+0000 7f4e3ffff700 1 -- 172.21.15.92:0/1696942763 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c00e040 con 0x7f4e580fc7b0 2023-06-23T01:05:25.073 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.066+0000 7f4e5de4a700 1 -- 172.21.15.92:0/1696942763 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e58114870 con 0x7f4e580fc7b0 2023-06-23T01:05:25.074 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.066+0000 7f4e3ffff700 1 -- 172.21.15.92:0/1696942763 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e4c009740 con 0x7f4e580fc7b0 2023-06-23T01:05:25.074 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.066+0000 7f4e3ffff700 1 -- 172.21.15.92:0/1696942763 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c013d30 con 0x7f4e580fc7b0 2023-06-23T01:05:25.074 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.066+0000 7f4e5de4a700 1 -- 172.21.15.92:0/1696942763 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e58114da0 con 0x7f4e580fc7b0 2023-06-23T01:05:25.075 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.067+0000 7f4e5de4a700 1 -- 172.21.15.92:0/1696942763 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e58103170 con 0x7f4e580fc7b0 2023-06-23T01:05:25.075 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.068+0000 7f4e3ffff700 1 -- 172.21.15.92:0/1696942763 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f4e4c013520 con 0x7f4e580fc7b0 2023-06-23T01:05:25.075 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.068+0000 7f4e3ffff700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4e4009b680 0x7f4e4009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:25.076 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.068+0000 7f4e3ffff700 1 -- 172.21.15.92:0/1696942763 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4961+0+0 (secure 0 0 0) 0x7f4e4c0c53e0 con 0x7f4e580fc7b0 2023-06-23T01:05:25.076 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.068+0000 7f4e577fe700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4e4009b680 0x7f4e4009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:25.076 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.069+0000 7f4e577fe700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4e4009b680 0x7f4e4009db30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e48007fd0 tx=0x7f4e48009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:05:25.077 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.072+0000 7f4e3ffff700 1 -- 172.21.15.92:0/1696942763 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f4e4c091950 con 0x7f4e580fc7b0 2023-06-23T01:05:25.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:24 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:05:25.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:24 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:05:25.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:24 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi046 2023-06-23T01:05:25.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:24 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi046 to 5724M 2023-06-23T01:05:25.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:24 smithi092 ceph-mon[85670]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:25.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:24 smithi092 ceph-mon[85670]: osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790] boot 2023-06-23T01:05:25.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:24 smithi092 ceph-mon[85670]: osdmap e45: 7 total, 7 up, 7 in 2023-06-23T01:05:25.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:24 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:25.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:24 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:25.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:24 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:05:25.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:24 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:05:25.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:24 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi046 2023-06-23T01:05:25.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:24 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi046 to 5724M 2023-06-23T01:05:25.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:24 smithi131 ceph-mon[86495]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:25.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:24 smithi131 ceph-mon[86495]: osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790] boot 2023-06-23T01:05:25.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:24 smithi131 ceph-mon[86495]: osdmap e45: 7 total, 7 up, 7 in 2023-06-23T01:05:25.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:24 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:25.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:24 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:25.269 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:25.266+0000 7f4e5de4a700 1 -- 172.21.15.92:0/1696942763 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e5802b140 con 0x7f4e4009b680 2023-06-23T01:05:25.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:24 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:05:25.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:24 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:05:25.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:24 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi046 2023-06-23T01:05:25.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:24 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi046 to 5724M 2023-06-23T01:05:25.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:24 smithi046 ceph-mon[84949]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-06-23T01:05:25.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:24 smithi046 ceph-mon[84949]: osd.6 [v2:172.21.15.46:6818/558288790,v1:172.21.15.46:6819/558288790] boot 2023-06-23T01:05:25.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:24 smithi046 ceph-mon[84949]: osdmap e45: 7 total, 7 up, 7 in 2023-06-23T01:05:25.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:25.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-06-23T01:05:26.388 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:26 smithi092 ceph-mon[85670]: osdmap e46: 7 total, 7 up, 7 in 2023-06-23T01:05:26.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:26 smithi092 ceph-mon[85670]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:05:26.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:26 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:05:26.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:26 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:05:26.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:26 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:26.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:26 smithi092 ceph-mon[85670]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:26.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:26 smithi131 ceph-mon[86495]: osdmap e46: 7 total, 7 up, 7 in 2023-06-23T01:05:26.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:26 smithi131 ceph-mon[86495]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:05:26.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:26 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:05:26.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:26 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:05:26.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:26 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:26.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:26 smithi131 ceph-mon[86495]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:26.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:26 smithi046 ceph-mon[84949]: osdmap e46: 7 total, 7 up, 7 in 2023-06-23T01:05:26.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:26 smithi046 ceph-mon[84949]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:05:26.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:26 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:05:26.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:26 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:05:26.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:26 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:26.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:26 smithi046 ceph-mon[84949]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 39 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:28.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:28 smithi131 ceph-mon[86495]: from='client.? 172.21.15.92:0/4088337733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb43609-8921-4891-b0c4-af932bfb69bb"}]: dispatch 2023-06-23T01:05:28.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:28 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb43609-8921-4891-b0c4-af932bfb69bb"}]: dispatch 2023-06-23T01:05:28.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:28 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2eb43609-8921-4891-b0c4-af932bfb69bb"}]': finished 2023-06-23T01:05:28.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:28 smithi131 ceph-mon[86495]: osdmap e47: 8 total, 7 up, 8 in 2023-06-23T01:05:28.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:28.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:28 smithi131 ceph-mon[86495]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:28.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:28 smithi131 ceph-mon[86495]: from='client.? 172.21.15.92:0/3280346931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:05:28.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:28 smithi046 ceph-mon[84949]: from='client.? 172.21.15.92:0/4088337733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb43609-8921-4891-b0c4-af932bfb69bb"}]: dispatch 2023-06-23T01:05:28.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:28 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb43609-8921-4891-b0c4-af932bfb69bb"}]: dispatch 2023-06-23T01:05:28.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:28 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2eb43609-8921-4891-b0c4-af932bfb69bb"}]': finished 2023-06-23T01:05:28.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:28 smithi046 ceph-mon[84949]: osdmap e47: 8 total, 7 up, 8 in 2023-06-23T01:05:28.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:28.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:28 smithi046 ceph-mon[84949]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:28.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:28 smithi046 ceph-mon[84949]: from='client.? 172.21.15.92:0/3280346931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:05:28.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:28 smithi092 ceph-mon[85670]: from='client.? 172.21.15.92:0/4088337733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb43609-8921-4891-b0c4-af932bfb69bb"}]: dispatch 2023-06-23T01:05:28.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:28 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb43609-8921-4891-b0c4-af932bfb69bb"}]: dispatch 2023-06-23T01:05:28.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:28 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2eb43609-8921-4891-b0c4-af932bfb69bb"}]': finished 2023-06-23T01:05:28.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:28 smithi092 ceph-mon[85670]: osdmap e47: 8 total, 7 up, 8 in 2023-06-23T01:05:28.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:28.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:28 smithi092 ceph-mon[85670]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:28.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:28 smithi092 ceph-mon[85670]: from='client.? 172.21.15.92:0/3280346931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:05:31.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:30 smithi092 ceph-mon[85670]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:31.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:30 smithi131 ceph-mon[86495]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:31.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:30 smithi046 ceph-mon[84949]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:33.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:32 smithi092 ceph-mon[85670]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:33.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:32 smithi131 ceph-mon[86495]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:33.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:32 smithi046 ceph-mon[84949]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:35.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:34 smithi092 ceph-mon[85670]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:35.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-06-23T01:05:35.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:35.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:34 smithi092 ceph-mon[85670]: Deploying daemon osd.7 on smithi092 2023-06-23T01:05:35.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:34 smithi131 ceph-mon[86495]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:35.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-06-23T01:05:35.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:35.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:34 smithi131 ceph-mon[86495]: Deploying daemon osd.7 on smithi092 2023-06-23T01:05:35.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:34 smithi046 ceph-mon[84949]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:35.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-06-23T01:05:35.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:35.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:34 smithi046 ceph-mon[84949]: Deploying daemon osd.7 on smithi092 2023-06-23T01:05:37.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:36 smithi092 ceph-mon[85670]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:37.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:36 smithi131 ceph-mon[86495]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:37.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:36 smithi046 ceph-mon[84949]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:38.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:38 smithi131 ceph-mon[86495]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:38.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:38 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:38.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:38 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:38.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:38 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:38.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:38 smithi046 ceph-mon[84949]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:38.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:38 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:38.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:38 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:38.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:38 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:38.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:38 smithi092 ceph-mon[85670]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:38.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:38.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:38.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:40.980 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.976+0000 7f4e3ffff700 1 -- 172.21.15.92:0/1696942763 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4e5802b140 con 0x7f4e4009b680 2023-06-23T01:05:40.980 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 7 on host 'smithi092' 2023-06-23T01:05:40.981 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 -- 172.21.15.92:0/1696942763 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4e4009b680 msgr2=0x7f4e4009db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:40.982 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4e4009b680 0x7f4e4009db30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4e48007fd0 tx=0x7f4e48009040 comp rx=0 tx=0).stop 2023-06-23T01:05:40.982 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 -- 172.21.15.92:0/1696942763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e580fc7b0 msgr2=0x7f4e5810cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:40.982 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e580fc7b0 0x7f4e5810cc20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c009f00 tx=0x7f4e4c009e30 comp rx=0 tx=0).stop 2023-06-23T01:05:40.982 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 -- 172.21.15.92:0/1696942763 shutdown_connections 2023-06-23T01:05:40.982 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e580fd0d0 0x7f4e58113cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:40.982 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4e580fbe20 0x7f4e5810c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:40.983 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4e4009b680 0x7f4e4009db30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:40.983 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 --2- 172.21.15.92:0/1696942763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4e580fc7b0 0x7f4e5810cc20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:40.983 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 -- 172.21.15.92:0/1696942763 >> 172.21.15.92:0/1696942763 conn(0x7f4e580f6a00 msgr2=0x7f4e581002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:05:40.983 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 -- 172.21.15.92:0/1696942763 shutdown_connections 2023-06-23T01:05:40.983 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:05:40.978+0000 7f4e5de4a700 1 -- 172.21.15.92:0/1696942763 wait complete. 2023-06-23T01:05:41.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:40 smithi092 ceph-mon[85670]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:41.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:40 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:41.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:40 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:41.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:40 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:41.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:40 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:41.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:40 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:41.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:40 smithi131 ceph-mon[86495]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:41.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:40 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:41.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:40 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:41.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:40 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:41.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:40 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:41.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:40 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:41.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:40 smithi046 ceph-mon[84949]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:41.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:40 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:41.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:40 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:41.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:40 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:41.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:40 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:41.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:40 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:41.426 DEBUG:teuthology.orchestra.run.smithi092:osd.7> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.7.service 2023-06-23T01:05:41.429 INFO:tasks.cephadm:Deploying osd.8 on smithi131 with /dev/vg_nvme/lv_2... 2023-06-23T01:05:41.429 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-06-23T01:05:41.457 INFO:journalctl@ceph.osd.7.smithi092.stdout:-- Logs begin at Fri 2023-06-23 00:44:08 UTC. -- 2023-06-23T01:05:42.021 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:05:42.123 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:41 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:42.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:41 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:42.124 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:41 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:42.215 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:41 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:42.215 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:41 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:42.215 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:41 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:42.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:41 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:42.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:41 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:42.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:41 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:05:42.389 INFO:journalctl@ceph.osd.7.smithi092.stdout:Jun 23 01:05:42 smithi092 conmon[106130]: 2023-06-23T01:05:42.119+0000 7f5769299540 -1 osd.7 0 log_to_monitors true 2023-06-23T01:05:43.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:42 smithi131 ceph-mon[86495]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:43.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:42 smithi131 ceph-mon[86495]: from='osd.7 [v2:172.21.15.92:6816/3796540277,v1:172.21.15.92:6817/3796540277]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-23T01:05:43.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:42 smithi131 ceph-mon[86495]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-23T01:05:43.233 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:42 smithi092 ceph-mon[85670]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:43.233 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:42 smithi092 ceph-mon[85670]: from='osd.7 [v2:172.21.15.92:6816/3796540277,v1:172.21.15.92:6817/3796540277]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-23T01:05:43.233 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:42 smithi092 ceph-mon[85670]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-23T01:05:43.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:42 smithi046 ceph-mon[84949]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:43.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:42 smithi046 ceph-mon[84949]: from='osd.7 [v2:172.21.15.92:6816/3796540277,v1:172.21.15.92:6817/3796540277]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-23T01:05:43.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:42 smithi046 ceph-mon[84949]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-06-23T01:05:43.526 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-06-23T01:05:43.526 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-06-23T01:05:43.526 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2023-06-23T01:05:43.527 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2023-06-23T01:05:43.527 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011435 s, 917 MB/s 2023-06-23T01:05:43.527 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2023-06-23T01:05:43.864 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi131:vg_nvme/lv_2 2023-06-23T01:05:44.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-06-23T01:05:44.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: osdmap e48: 8 total, 7 up, 8 in 2023-06-23T01:05:44.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:44.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='osd.7 [v2:172.21.15.92:6816/3796540277,v1:172.21.15.92:6817/3796540277]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:05:44.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:05:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:44.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:44.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:43 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-06-23T01:05:44.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: osdmap e48: 8 total, 7 up, 8 in 2023-06-23T01:05:44.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='osd.7 [v2:172.21.15.92:6816/3796540277,v1:172.21.15.92:6817/3796540277]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:05:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:05:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:44.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:44.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:43 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.389 INFO:journalctl@ceph.osd.7.smithi092.stdout:Jun 23 01:05:43 smithi092 conmon[106130]: 2023-06-23T01:05:43.971+0000 7f575fa7e700 -1 osd.7 0 waiting for initial osdmap 2023-06-23T01:05:44.390 INFO:journalctl@ceph.osd.7.smithi092.stdout:Jun 23 01:05:43 smithi092 conmon[106130]: 2023-06-23T01:05:43.981+0000 7f575b07a700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:05:44.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-06-23T01:05:44.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: osdmap e48: 8 total, 7 up, 8 in 2023-06-23T01:05:44.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:44.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='osd.7 [v2:172.21.15.92:6816/3796540277,v1:172.21.15.92:6817/3796540277]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:05:44.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:05:44.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:44.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:44.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:05:44.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:44.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:05:44.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:43 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:05:44.451 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:05:44.976 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:44 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi092 2023-06-23T01:05:44.976 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:44 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi092 to 5722M 2023-06-23T01:05:44.976 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:44 smithi131 ceph-mon[86495]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:44.977 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:44 smithi131 ceph-mon[86495]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:05:44.977 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:44 smithi131 ceph-mon[86495]: osdmap e49: 8 total, 7 up, 8 in 2023-06-23T01:05:44.977 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:44.977 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:45.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:44 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi092 2023-06-23T01:05:45.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:44 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi092 to 5722M 2023-06-23T01:05:45.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:44 smithi046 ceph-mon[84949]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:45.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:44 smithi046 ceph-mon[84949]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:05:45.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:44 smithi046 ceph-mon[84949]: osdmap e49: 8 total, 7 up, 8 in 2023-06-23T01:05:45.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:45.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:45.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:44 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi092 2023-06-23T01:05:45.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:44 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi092 to 5722M 2023-06-23T01:05:45.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:44 smithi092 ceph-mon[85670]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:45.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:44 smithi092 ceph-mon[85670]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:05:45.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:44 smithi092 ceph-mon[85670]: osdmap e49: 8 total, 7 up, 8 in 2023-06-23T01:05:45.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:45.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:45.654 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 -- 172.21.15.131:0/849343361 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc0fe7e0 msgr2=0x7f42fc1050a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:45.654 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 --2- 172.21.15.131:0/849343361 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc0fe7e0 0x7f42fc1050a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009d60 tx=0x7f42f4004d10 comp rx=0 tx=0).stop 2023-06-23T01:05:45.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 -- 172.21.15.131:0/849343361 shutdown_connections 2023-06-23T01:05:45.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 --2- 172.21.15.131:0/849343361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42fc0fcc40 0x7f42fc0fd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:45.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 --2- 172.21.15.131:0/849343361 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc0fe7e0 0x7f42fc1050a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:45.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 --2- 172.21.15.131:0/849343361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0fde50 0x7f42fc0fe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:45.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 -- 172.21.15.131:0/849343361 >> 172.21.15.131:0/849343361 conn(0x7f42fc0f83c0 msgr2=0x7f42fc0fa7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:05:45.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 -- 172.21.15.131:0/849343361 shutdown_connections 2023-06-23T01:05:45.656 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 -- 172.21.15.131:0/849343361 wait complete. 2023-06-23T01:05:45.656 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 Processor -- start 2023-06-23T01:05:45.656 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 -- start start 2023-06-23T01:05:45.656 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0fcc40 0x7f42fc10d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:45.656 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42fc0fde50 0x7f42fc10daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:45.656 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc0fe7e0 0x7f42fc114b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:45.656 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc115190 con 0x7f42fc0fcc40 2023-06-23T01:05:45.657 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc115300 con 0x7f42fc0fde50 2023-06-23T01:05:45.657 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.651+0000 7f430425b700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc1154c0 con 0x7f42fc0fe7e0 2023-06-23T01:05:45.658 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f43027f8700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc0fe7e0 0x7f42fc114b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:45.659 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f43027f8700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc0fe7e0 0x7f42fc114b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:49850/0 (socket says 172.21.15.131:49850) 2023-06-23T01:05:45.659 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f43027f8700 1 -- 172.21.15.131:0/2008060354 learned_addr learned my addr 172.21.15.131:0/2008060354 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-06-23T01:05:45.659 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f43017f6700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42fc0fde50 0x7f42fc10daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:45.659 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f4301ff7700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0fcc40 0x7f42fc10d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:45.659 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f43027f8700 1 -- 172.21.15.131:0/2008060354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42fc0fde50 msgr2=0x7f42fc10daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:45.659 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f43027f8700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42fc0fde50 0x7f42fc10daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:45.660 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f43027f8700 1 -- 172.21.15.131:0/2008060354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0fcc40 msgr2=0x7f42fc10d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:05:45.660 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f43027f8700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0fcc40 0x7f42fc10d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:05:45.660 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f43027f8700 1 -- 172.21.15.131:0/2008060354 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f4009940 con 0x7f42fc0fe7e0 2023-06-23T01:05:45.660 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f4301ff7700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0fcc40 0x7f42fc10d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:05:45.660 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f43027f8700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc0fe7e0 0x7f42fc114b80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f42f4015040 tx=0x7f42f4004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:05:45.660 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f42f27fc700 1 -- 172.21.15.131:0/2008060354 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f400e030 con 0x7f42fc0fe7e0 2023-06-23T01:05:45.660 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f430425b700 1 -- 172.21.15.131:0/2008060354 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42fc115740 con 0x7f42fc0fe7e0 2023-06-23T01:05:45.661 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f42f27fc700 1 -- 172.21.15.131:0/2008060354 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f42f401caf0 con 0x7f42fc0fe7e0 2023-06-23T01:05:45.661 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.652+0000 7f42f27fc700 1 -- 172.21.15.131:0/2008060354 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f4012830 con 0x7f42fc0fe7e0 2023-06-23T01:05:45.661 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.653+0000 7f430425b700 1 -- 172.21.15.131:0/2008060354 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42fc115bf0 con 0x7f42fc0fe7e0 2023-06-23T01:05:45.661 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.653+0000 7f430425b700 1 -- 172.21.15.131:0/2008060354 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42fc103fe0 con 0x7f42fc0fe7e0 2023-06-23T01:05:45.661 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.654+0000 7f42f27fc700 1 -- 172.21.15.131:0/2008060354 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f42f401cc60 con 0x7f42fc0fe7e0 2023-06-23T01:05:45.661 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.655+0000 7f42f27fc700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f42e809b680 0x7f42e809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:05:45.661 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.655+0000 7f42f27fc700 1 -- 172.21.15.131:0/2008060354 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f42f40c6930 con 0x7f42fc0fe7e0 2023-06-23T01:05:45.662 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.655+0000 7f4301ff7700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f42e809b680 0x7f42e809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:05:45.662 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.656+0000 7f4301ff7700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f42e809b680 0x7f42e809db30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f42f8007fd0 tx=0x7f42f8009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:05:45.662 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.659+0000 7f42f27fc700 1 -- 172.21.15.131:0/2008060354 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f42f4092cd0 con 0x7f42fc0fe7e0 2023-06-23T01:05:45.855 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:05:45.852+0000 7f430425b700 1 -- 172.21.15.131:0/2008060354 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f42fc02b0d0 con 0x7f42e809b680 2023-06-23T01:05:46.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:45 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:05:46.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:45 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:05:46.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:45 smithi131 ceph-mon[86495]: osd.7 [v2:172.21.15.92:6816/3796540277,v1:172.21.15.92:6817/3796540277] boot 2023-06-23T01:05:46.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:45 smithi131 ceph-mon[86495]: osdmap e50: 8 total, 8 up, 8 in 2023-06-23T01:05:46.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:45 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:46.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:45 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:05:46.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:45 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:05:46.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:45 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:46.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:45 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:05:46.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:45 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:05:46.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:45 smithi046 ceph-mon[84949]: osd.7 [v2:172.21.15.92:6816/3796540277,v1:172.21.15.92:6817/3796540277] boot 2023-06-23T01:05:46.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:45 smithi046 ceph-mon[84949]: osdmap e50: 8 total, 8 up, 8 in 2023-06-23T01:05:46.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:45 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:46.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:45 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:05:46.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:45 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:05:46.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:45 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:46.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:45 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:05:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:45 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:05:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:45 smithi092 ceph-mon[85670]: osd.7 [v2:172.21.15.92:6816/3796540277,v1:172.21.15.92:6817/3796540277] boot 2023-06-23T01:05:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:45 smithi092 ceph-mon[85670]: osdmap e50: 8 total, 8 up, 8 in 2023-06-23T01:05:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-06-23T01:05:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:05:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:05:46.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:45 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:47.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:46 smithi046 ceph-mon[84949]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:47.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:46 smithi046 ceph-mon[84949]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:05:47.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:46 smithi046 ceph-mon[84949]: osdmap e51: 8 total, 8 up, 8 in 2023-06-23T01:05:47.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:46 smithi092 ceph-mon[85670]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:47.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:46 smithi092 ceph-mon[85670]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:05:47.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:46 smithi092 ceph-mon[85670]: osdmap e51: 8 total, 8 up, 8 in 2023-06-23T01:05:47.424 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:46 smithi131 ceph-mon[86495]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-06-23T01:05:47.424 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:46 smithi131 ceph-mon[86495]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:05:47.424 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:46 smithi131 ceph-mon[86495]: osdmap e51: 8 total, 8 up, 8 in 2023-06-23T01:05:48.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:47 smithi046 ceph-mon[84949]: osdmap e52: 8 total, 8 up, 8 in 2023-06-23T01:05:48.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:47 smithi046 ceph-mon[84949]: osdmap e53: 8 total, 8 up, 8 in 2023-06-23T01:05:48.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:47 smithi092 ceph-mon[85670]: osdmap e52: 8 total, 8 up, 8 in 2023-06-23T01:05:48.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:47 smithi092 ceph-mon[85670]: osdmap e53: 8 total, 8 up, 8 in 2023-06-23T01:05:48.481 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:47 smithi131 ceph-mon[86495]: osdmap e52: 8 total, 8 up, 8 in 2023-06-23T01:05:48.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:47 smithi131 ceph-mon[86495]: osdmap e53: 8 total, 8 up, 8 in 2023-06-23T01:05:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:48 smithi046 ceph-mon[84949]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:48 smithi046 ceph-mon[84949]: from='client.? 172.21.15.131:0/3810511131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0139b39e-f933-4685-b056-a6c27fa87fd7"}]: dispatch 2023-06-23T01:05:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:48 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0139b39e-f933-4685-b056-a6c27fa87fd7"}]: dispatch 2023-06-23T01:05:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:48 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0139b39e-f933-4685-b056-a6c27fa87fd7"}]': finished 2023-06-23T01:05:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:48 smithi046 ceph-mon[84949]: osdmap e54: 9 total, 8 up, 9 in 2023-06-23T01:05:49.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:48 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:05:49.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:48 smithi046 ceph-mon[84949]: from='client.? 172.21.15.131:0/4201364184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:05:49.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:48 smithi092 ceph-mon[85670]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:49.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:48 smithi092 ceph-mon[85670]: from='client.? 172.21.15.131:0/3810511131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0139b39e-f933-4685-b056-a6c27fa87fd7"}]: dispatch 2023-06-23T01:05:49.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:48 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0139b39e-f933-4685-b056-a6c27fa87fd7"}]: dispatch 2023-06-23T01:05:49.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:48 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0139b39e-f933-4685-b056-a6c27fa87fd7"}]': finished 2023-06-23T01:05:49.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:48 smithi092 ceph-mon[85670]: osdmap e54: 9 total, 8 up, 9 in 2023-06-23T01:05:49.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:48 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:05:49.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:48 smithi092 ceph-mon[85670]: from='client.? 172.21.15.131:0/4201364184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:05:49.481 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:48 smithi131 ceph-mon[86495]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:49.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:48 smithi131 ceph-mon[86495]: from='client.? 172.21.15.131:0/3810511131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0139b39e-f933-4685-b056-a6c27fa87fd7"}]: dispatch 2023-06-23T01:05:49.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:48 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0139b39e-f933-4685-b056-a6c27fa87fd7"}]: dispatch 2023-06-23T01:05:49.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:48 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0139b39e-f933-4685-b056-a6c27fa87fd7"}]': finished 2023-06-23T01:05:49.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:48 smithi131 ceph-mon[86495]: osdmap e54: 9 total, 8 up, 9 in 2023-06-23T01:05:49.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:48 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:05:49.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:48 smithi131 ceph-mon[86495]: from='client.? 172.21.15.131:0/4201364184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:05:51.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:50 smithi046 ceph-mon[84949]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:51.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:50 smithi092 ceph-mon[85670]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:51.481 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:50 smithi131 ceph-mon[86495]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:53.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:52 smithi131 ceph-mon[86495]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:53.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:52 smithi046 ceph-mon[84949]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:53.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:52 smithi092 ceph-mon[85670]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:55.245 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:54 smithi131 ceph-mon[86495]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:55.245 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:54 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-06-23T01:05:55.245 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:54 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:55.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:54 smithi046 ceph-mon[84949]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:55.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:54 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-06-23T01:05:55.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:54 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:55.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:54 smithi092 ceph-mon[85670]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:55.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:54 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-06-23T01:05:55.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:54 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:05:56.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:55 smithi046 ceph-mon[84949]: Deploying daemon osd.8 on smithi131 2023-06-23T01:05:56.384 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:55 smithi131 ceph-mon[86495]: Deploying daemon osd.8 on smithi131 2023-06-23T01:05:56.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:55 smithi092 ceph-mon[85670]: Deploying daemon osd.8 on smithi131 2023-06-23T01:05:57.128 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:56 smithi131 ceph-mon[86495]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:57.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:56 smithi046 ceph-mon[84949]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:57.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:56 smithi092 ceph-mon[85670]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:58.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:58 smithi131 ceph-mon[86495]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:58.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:58 smithi046 ceph-mon[84949]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:05:58.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:58 smithi092 ceph-mon[85670]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:00.136 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:00.137 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:00.137 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:05:59 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:00.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:00.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:00.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:05:59 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:00.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:00.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:00.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:05:59 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:00.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:00 smithi131 ceph-mon[86495]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:01.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:00 smithi046 ceph-mon[84949]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:01.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:00 smithi092 ceph-mon[85670]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:01.567 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.564+0000 7f42f27fc700 1 -- 172.21.15.131:0/2008060354 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f42fc02b0d0 con 0x7f42e809b680 2023-06-23T01:06:01.568 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 8 on host 'smithi131' 2023-06-23T01:06:01.569 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.566+0000 7f430425b700 1 -- 172.21.15.131:0/2008060354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f42e809b680 msgr2=0x7f42e809db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:01.570 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.566+0000 7f430425b700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f42e809b680 0x7f42e809db30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f42f8007fd0 tx=0x7f42f8009040 comp rx=0 tx=0).stop 2023-06-23T01:06:01.570 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.566+0000 7f430425b700 1 -- 172.21.15.131:0/2008060354 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc0fe7e0 msgr2=0x7f42fc114b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:01.570 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.566+0000 7f430425b700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc0fe7e0 0x7f42fc114b80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f42f4015040 tx=0x7f42f4004d10 comp rx=0 tx=0).stop 2023-06-23T01:06:01.570 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.566+0000 7f430425b700 1 -- 172.21.15.131:0/2008060354 shutdown_connections 2023-06-23T01:06:01.570 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.567+0000 7f430425b700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f42fc0fe7e0 0x7f42fc114b80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:01.570 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.567+0000 7f430425b700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42fc0fcc40 0x7f42fc10d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:01.570 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.567+0000 7f430425b700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f42e809b680 0x7f42e809db30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:01.571 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.567+0000 7f430425b700 1 --2- 172.21.15.131:0/2008060354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f42fc0fde50 0x7f42fc10daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:01.571 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.567+0000 7f430425b700 1 -- 172.21.15.131:0/2008060354 >> 172.21.15.131:0/2008060354 conn(0x7f42fc0f83c0 msgr2=0x7f42fc101160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:06:01.571 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.567+0000 7f430425b700 1 -- 172.21.15.131:0/2008060354 shutdown_connections 2023-06-23T01:06:01.571 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:01.567+0000 7f430425b700 1 -- 172.21.15.131:0/2008060354 wait complete. 2023-06-23T01:06:01.999 DEBUG:teuthology.orchestra.run.smithi131:osd.8> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.8.service 2023-06-23T01:06:02.001 INFO:tasks.cephadm:Deploying osd.9 on smithi046 with /dev/vg_nvme/lv_1... 2023-06-23T01:06:02.002 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-06-23T01:06:02.028 INFO:journalctl@ceph.osd.8.smithi131.stdout:-- Logs begin at Fri 2023-06-23 00:44:02 UTC. -- 2023-06-23T01:06:02.614 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:06:02.816 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.816 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.816 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.816 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.816 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:02 smithi046 ceph-mon[84949]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:02.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:02 smithi092 ceph-mon[85670]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:02.903 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.903 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.903 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.903 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:02.904 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:02 smithi131 ceph-mon[86495]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:03.232 INFO:journalctl@ceph.osd.8.smithi131.stdout:Jun 23 01:06:02 smithi131 conmon[108138]: 2023-06-23T01:06:02.921+0000 7f43a19c9540 -1 osd.8 0 log_to_monitors true 2023-06-23T01:06:03.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:03 smithi046 ceph-mon[84949]: from='osd.8 [v2:172.21.15.131:6816/1004740496,v1:172.21.15.131:6817/1004740496]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-23T01:06:03.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:03 smithi046 ceph-mon[84949]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-23T01:06:03.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:03 smithi131 ceph-mon[86495]: from='osd.8 [v2:172.21.15.131:6816/1004740496,v1:172.21.15.131:6817/1004740496]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-23T01:06:03.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:03 smithi131 ceph-mon[86495]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-23T01:06:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-06-23T01:06:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-06-23T01:06:04.026 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2023-06-23T01:06:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2023-06-23T01:06:04.026 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00878652 s, 1.2 GB/s 2023-06-23T01:06:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2023-06-23T01:06:04.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:03 smithi092 ceph-mon[85670]: from='osd.8 [v2:172.21.15.131:6816/1004740496,v1:172.21.15.131:6817/1004740496]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-23T01:06:04.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:03 smithi092 ceph-mon[85670]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-06-23T01:06:04.551 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi046:vg_nvme/lv_1 2023-06-23T01:06:04.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-06-23T01:06:04.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: osdmap e55: 9 total, 8 up, 9 in 2023-06-23T01:06:04.664 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:04.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='osd.8 [v2:172.21.15.131:6816/1004740496,v1:172.21.15.131:6817/1004740496]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:06:04.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:06:04.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:04.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:04.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:04.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:04.665 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:04.666 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:04.666 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:04.666 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:04.666 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:04.666 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:04.666 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:04.666 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:04.667 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:04.667 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:04.982 INFO:journalctl@ceph.osd.8.smithi131.stdout:Jun 23 01:06:04 smithi131 conmon[108138]: 2023-06-23T01:06:04.663+0000 7f43981ae700 -1 osd.8 0 waiting for initial osdmap 2023-06-23T01:06:04.982 INFO:journalctl@ceph.osd.8.smithi131.stdout:Jun 23 01:06:04 smithi131 conmon[108138]: 2023-06-23T01:06:04.673+0000 7f4390fa5700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:06:04.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-06-23T01:06:04.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: osdmap e55: 9 total, 8 up, 9 in 2023-06-23T01:06:04.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:04.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='osd.8 [v2:172.21.15.131:6816/1004740496,v1:172.21.15.131:6817/1004740496]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:06:04.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:06:04.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:04.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:04.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:04.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:04.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:04.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:04.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:04.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:04.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:04.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:04.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:04.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:04.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:04.985 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:05.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-06-23T01:06:05.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: osdmap e55: 9 total, 8 up, 9 in 2023-06-23T01:06:05.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:05.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='osd.8 [v2:172.21.15.131:6816/1004740496,v1:172.21.15.131:6817/1004740496]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:06:05.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:06:05.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:05.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:05.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:05.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:05.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:05.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:05.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:05.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:05.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:05.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:05.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:05.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:05.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:05.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:05.145 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:06:05.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:05 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi131 2023-06-23T01:06:05.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:05 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi131 to 5722M 2023-06-23T01:06:05.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:05 smithi131 ceph-mon[86495]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:06:05.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:05 smithi131 ceph-mon[86495]: osdmap e56: 9 total, 8 up, 9 in 2023-06-23T01:06:05.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:05 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:05.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:05 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:05.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:05 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:06.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:05 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi131 2023-06-23T01:06:06.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:05 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi131 to 5722M 2023-06-23T01:06:06.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:05 smithi046 ceph-mon[84949]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:06:06.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:05 smithi046 ceph-mon[84949]: osdmap e56: 9 total, 8 up, 9 in 2023-06-23T01:06:06.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:05 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:06.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:05 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:06.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:05 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:06.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:05 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi131 2023-06-23T01:06:06.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:05 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi131 to 5722M 2023-06-23T01:06:06.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:05 smithi092 ceph-mon[85670]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:06:06.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:05 smithi092 ceph-mon[85670]: osdmap e56: 9 total, 8 up, 9 in 2023-06-23T01:06:06.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:05 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:06.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:05 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:06.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:05 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:06.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.337+0000 7f9714f0a700 1 -- 172.21.15.46:0/4191531005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97100fde60 msgr2=0x7f97100fe2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:06.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.337+0000 7f9714f0a700 1 --2- 172.21.15.46:0/4191531005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97100fde60 0x7f97100fe2b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f97040049c0 tx=0x7f9704009e30 comp rx=0 tx=0).stop 2023-06-23T01:06:06.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.337+0000 7f9714f0a700 1 -- 172.21.15.46:0/4191531005 shutdown_connections 2023-06-23T01:06:06.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.337+0000 7f9714f0a700 1 --2- 172.21.15.46:0/4191531005 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97100fcc50 0x7f97100fd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.337+0000 7f9714f0a700 1 --2- 172.21.15.46:0/4191531005 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97100fe7f0 0x7f97101050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.337+0000 7f9714f0a700 1 --2- 172.21.15.46:0/4191531005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97100fde60 0x7f97100fe2b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.337+0000 7f9714f0a700 1 -- 172.21.15.46:0/4191531005 >> 172.21.15.46:0/4191531005 conn(0x7f97100f83b0 msgr2=0x7f97100fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:06:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.337+0000 7f9714f0a700 1 -- 172.21.15.46:0/4191531005 shutdown_connections 2023-06-23T01:06:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.337+0000 7f9714f0a700 1 -- 172.21.15.46:0/4191531005 wait complete. 2023-06-23T01:06:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f9714f0a700 1 Processor -- start 2023-06-23T01:06:06.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f9714f0a700 1 -- start start 2023-06-23T01:06:06.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f9714f0a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97100fcc50 0x7f971010fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:06.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f9714f0a700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97100fde60 0x7f97101100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:06.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f9714f0a700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97100fe7f0 0x7f9710117150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:06.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f9714f0a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9710117760 con 0x7f97100fcc50 2023-06-23T01:06:06.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f9714f0a700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97101178d0 con 0x7f97100fe7f0 2023-06-23T01:06:06.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f9714f0a700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9710117a90 con 0x7f97100fde60 2023-06-23T01:06:06.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f970e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97100fcc50 0x7f971010fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:06:06.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f970dd9b700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97100fde60 0x7f97101100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:06:06.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f970e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97100fcc50 0x7f971010fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33742/0 (socket says 172.21.15.46:33742) 2023-06-23T01:06:06.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f970e59c700 1 -- 172.21.15.46:0/979751972 learned_addr learned my addr 172.21.15.46:0/979751972 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:06:06.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.338+0000 7f970ed9d700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97100fe7f0 0x7f9710117150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:06:06.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.339+0000 7f970e59c700 1 -- 172.21.15.46:0/979751972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97100fde60 msgr2=0x7f97101100c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:06.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.339+0000 7f970e59c700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97100fde60 0x7f97101100c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:06.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.339+0000 7f970e59c700 1 -- 172.21.15.46:0/979751972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97100fe7f0 msgr2=0x7f9710117150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:06.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.339+0000 7f970e59c700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97100fe7f0 0x7f9710117150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:06.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.339+0000 7f970e59c700 1 -- 172.21.15.46:0/979751972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9704005040 con 0x7f97100fcc50 2023-06-23T01:06:06.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.339+0000 7f970e59c700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97100fcc50 0x7f971010fb80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f970000b810 tx=0x7f9700009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:06:06.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.339+0000 7f96feffd700 1 -- 172.21.15.46:0/979751972 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970000f510 con 0x7f97100fcc50 2023-06-23T01:06:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.339+0000 7f96feffd700 1 -- 172.21.15.46:0/979751972 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f970000fb10 con 0x7f97100fcc50 2023-06-23T01:06:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.339+0000 7f96feffd700 1 -- 172.21.15.46:0/979751972 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700011b80 con 0x7f97100fcc50 2023-06-23T01:06:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.339+0000 7f9714f0a700 1 -- 172.21.15.46:0/979751972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9710117d70 con 0x7f97100fcc50 2023-06-23T01:06:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.339+0000 7f9714f0a700 1 -- 172.21.15.46:0/979751972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97101181b0 con 0x7f97100fcc50 2023-06-23T01:06:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.340+0000 7f9714f0a700 1 -- 172.21.15.46:0/979751972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9710103890 con 0x7f97100fcc50 2023-06-23T01:06:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.341+0000 7f96feffd700 1 -- 172.21.15.46:0/979751972 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f970000fc80 con 0x7f97100fcc50 2023-06-23T01:06:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.342+0000 7f96feffd700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f96f809b700 0x7f96f809dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.342+0000 7f970dd9b700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f96f809b700 0x7f96f809dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:06:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.342+0000 7f96feffd700 1 -- 172.21.15.46:0/979751972 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f97000c3e10 con 0x7f97100fcc50 2023-06-23T01:06:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.343+0000 7f970dd9b700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f96f809b700 0x7f96f809dbb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9704004f30 tx=0x7f9704009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:06:06.349 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.347+0000 7f96feffd700 1 -- 172.21.15.46:0/979751972 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f9700090000 con 0x7f97100fcc50 2023-06-23T01:06:06.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:06.549+0000 7f9714f0a700 1 -- 172.21.15.46:0/979751972 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f971002b430 con 0x7f96f809b700 2023-06-23T01:06:06.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:06 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:06:06.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:06 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:06:06.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:06 smithi131 ceph-mon[86495]: osd.8 [v2:172.21.15.131:6816/1004740496,v1:172.21.15.131:6817/1004740496] boot 2023-06-23T01:06:06.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:06 smithi131 ceph-mon[86495]: osdmap e57: 9 total, 9 up, 9 in 2023-06-23T01:06:06.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:06 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:06.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:06 smithi131 ceph-mon[86495]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:06.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:06 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:06:06.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:06 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:06:06.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:06 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:07.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:06 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:06:07.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:06 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:06:07.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:06 smithi046 ceph-mon[84949]: osd.8 [v2:172.21.15.131:6816/1004740496,v1:172.21.15.131:6817/1004740496] boot 2023-06-23T01:06:07.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:06 smithi046 ceph-mon[84949]: osdmap e57: 9 total, 9 up, 9 in 2023-06-23T01:06:07.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:07.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:06 smithi046 ceph-mon[84949]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:07.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:06:07.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:06:07.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:07.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:06 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:06:07.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:06 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:06:07.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:06 smithi092 ceph-mon[85670]: osd.8 [v2:172.21.15.131:6816/1004740496,v1:172.21.15.131:6817/1004740496] boot 2023-06-23T01:06:07.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:06 smithi092 ceph-mon[85670]: osdmap e57: 9 total, 9 up, 9 in 2023-06-23T01:06:07.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:06 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-06-23T01:06:07.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:06 smithi092 ceph-mon[85670]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-06-23T01:06:07.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:06 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:06:07.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:06 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:06:07.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:06 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:07.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:07 smithi131 ceph-mon[86495]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:06:07.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:07 smithi131 ceph-mon[86495]: osdmap e58: 9 total, 9 up, 9 in 2023-06-23T01:06:08.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:07 smithi046 ceph-mon[84949]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:06:08.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:07 smithi046 ceph-mon[84949]: osdmap e58: 9 total, 9 up, 9 in 2023-06-23T01:06:08.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:07 smithi092 ceph-mon[85670]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:06:08.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:07 smithi092 ceph-mon[85670]: osdmap e58: 9 total, 9 up, 9 in 2023-06-23T01:06:08.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:08 smithi131 ceph-mon[86495]: osdmap e59: 9 total, 9 up, 9 in 2023-06-23T01:06:08.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:08 smithi131 ceph-mon[86495]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:09.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:08 smithi046 ceph-mon[84949]: osdmap e59: 9 total, 9 up, 9 in 2023-06-23T01:06:09.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:08 smithi046 ceph-mon[84949]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:09.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:08 smithi092 ceph-mon[85670]: osdmap e59: 9 total, 9 up, 9 in 2023-06-23T01:06:09.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:08 smithi092 ceph-mon[85670]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:09.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:09 smithi131 ceph-mon[86495]: osdmap e60: 9 total, 9 up, 9 in 2023-06-23T01:06:09.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:09 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/172542118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bd083b4-f766-4e1b-b959-19d2fa82d77f"}]: dispatch 2023-06-23T01:06:09.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:09 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/172542118' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bd083b4-f766-4e1b-b959-19d2fa82d77f"}]': finished 2023-06-23T01:06:09.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:09 smithi131 ceph-mon[86495]: osdmap e61: 10 total, 9 up, 10 in 2023-06-23T01:06:09.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:09 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:09.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:09 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/432517956' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:06:10.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:09 smithi046 ceph-mon[84949]: osdmap e60: 9 total, 9 up, 9 in 2023-06-23T01:06:10.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:09 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/172542118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bd083b4-f766-4e1b-b959-19d2fa82d77f"}]: dispatch 2023-06-23T01:06:10.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:09 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/172542118' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bd083b4-f766-4e1b-b959-19d2fa82d77f"}]': finished 2023-06-23T01:06:10.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:09 smithi046 ceph-mon[84949]: osdmap e61: 10 total, 9 up, 10 in 2023-06-23T01:06:10.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:09 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:10.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:09 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/432517956' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:06:10.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:09 smithi092 ceph-mon[85670]: osdmap e60: 9 total, 9 up, 9 in 2023-06-23T01:06:10.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:09 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/172542118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bd083b4-f766-4e1b-b959-19d2fa82d77f"}]: dispatch 2023-06-23T01:06:10.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:09 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/172542118' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bd083b4-f766-4e1b-b959-19d2fa82d77f"}]': finished 2023-06-23T01:06:10.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:09 smithi092 ceph-mon[85670]: osdmap e61: 10 total, 9 up, 10 in 2023-06-23T01:06:10.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:09 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:10.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:09 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/432517956' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:06:10.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:10 smithi131 ceph-mon[86495]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:11.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:10 smithi046 ceph-mon[84949]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:11.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:10 smithi092 ceph-mon[85670]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:13.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:12 smithi131 ceph-mon[86495]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 112 KiB/s, 0 objects/s recovering 2023-06-23T01:06:13.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:12 smithi046 ceph-mon[84949]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 112 KiB/s, 0 objects/s recovering 2023-06-23T01:06:13.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:12 smithi092 ceph-mon[85670]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 112 KiB/s, 0 objects/s recovering 2023-06-23T01:06:15.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:14 smithi131 ceph-mon[86495]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-06-23T01:06:15.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:14 smithi046 ceph-mon[84949]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-06-23T01:06:15.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:14 smithi092 ceph-mon[85670]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2023-06-23T01:06:17.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:16 smithi131 ceph-mon[86495]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:06:17.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-06-23T01:06:17.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:16 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:17.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:16 smithi131 ceph-mon[86495]: Deploying daemon osd.9 on smithi046 2023-06-23T01:06:17.284 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:16 smithi046 ceph-mon[84949]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:06:17.284 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-06-23T01:06:17.284 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:16 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:17.284 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:16 smithi046 ceph-mon[84949]: Deploying daemon osd.9 on smithi046 2023-06-23T01:06:17.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:16 smithi092 ceph-mon[85670]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:06:17.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-06-23T01:06:17.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:16 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:17.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:16 smithi092 ceph-mon[85670]: Deploying daemon osd.9 on smithi046 2023-06-23T01:06:18.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:18 smithi046 ceph-mon[84949]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-06-23T01:06:18.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:18 smithi131 ceph-mon[86495]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-06-23T01:06:18.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:18 smithi092 ceph-mon[85670]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-06-23T01:06:21.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:20 smithi046 ceph-mon[84949]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-06-23T01:06:21.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:20 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:21.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:20 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:21.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:20 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:21.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:20 smithi131 ceph-mon[86495]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-06-23T01:06:21.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:20 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:21.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:20 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:21.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:20 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:21.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:20 smithi092 ceph-mon[85670]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-06-23T01:06:21.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:20 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:21.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:20 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:21.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:20 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:23.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:22 smithi046 ceph-mon[84949]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:06:23.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.147+0000 7f96feffd700 1 -- 172.21.15.46:0/979751972 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f971002b430 con 0x7f96f809b700 2023-06-23T01:06:23.150 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 9 on host 'smithi046' 2023-06-23T01:06:23.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.149+0000 7f9714f0a700 1 -- 172.21.15.46:0/979751972 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f96f809b700 msgr2=0x7f96f809dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:23.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.149+0000 7f9714f0a700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f96f809b700 0x7f96f809dbb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9704004f30 tx=0x7f9704009e30 comp rx=0 tx=0).stop 2023-06-23T01:06:23.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.149+0000 7f9714f0a700 1 -- 172.21.15.46:0/979751972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97100fcc50 msgr2=0x7f971010fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:23.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.149+0000 7f9714f0a700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97100fcc50 0x7f971010fb80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f970000b810 tx=0x7f9700009220 comp rx=0 tx=0).stop 2023-06-23T01:06:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.149+0000 7f9714f0a700 1 -- 172.21.15.46:0/979751972 shutdown_connections 2023-06-23T01:06:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.149+0000 7f9714f0a700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f96f809b700 0x7f96f809dbb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.149+0000 7f9714f0a700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97100fe7f0 0x7f9710117150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.149+0000 7f9714f0a700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97100fcc50 0x7f971010fb80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.149+0000 7f9714f0a700 1 --2- 172.21.15.46:0/979751972 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97100fde60 0x7f97101100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.149+0000 7f9714f0a700 1 -- 172.21.15.46:0/979751972 >> 172.21.15.46:0/979751972 conn(0x7f97100f83b0 msgr2=0x7f97100fa320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:06:23.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.150+0000 7f9714f0a700 1 -- 172.21.15.46:0/979751972 shutdown_connections 2023-06-23T01:06:23.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:06:23.150+0000 7f9714f0a700 1 -- 172.21.15.46:0/979751972 wait complete. 2023-06-23T01:06:23.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:22 smithi131 ceph-mon[86495]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:06:23.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:22 smithi092 ceph-mon[85670]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-06-23T01:06:23.613 DEBUG:teuthology.orchestra.run.smithi046:osd.9> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.9.service 2023-06-23T01:06:23.615 INFO:tasks.cephadm:Deploying osd.10 on smithi092 with /dev/vg_nvme/lv_1... 2023-06-23T01:06:23.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-06-23T01:06:23.643 INFO:journalctl@ceph.osd.9.smithi046.stdout:-- Logs begin at Fri 2023-06-23 00:44:15 UTC. -- 2023-06-23T01:06:24.210 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-23T01:06:24.451 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:24 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.451 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:24 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.451 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:24 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:24 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.452 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:24 smithi092 ceph-mon[85670]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:24.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:24 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:24 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:24 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:24 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:24 smithi131 ceph-mon[86495]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:24.568 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:06:24 smithi046 conmon[123172]: 2023-06-23T01:06:24.215+0000 7f60ada7c540 -1 osd.9 0 log_to_monitors true 2023-06-23T01:06:24.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:24.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:24 smithi046 ceph-mon[84949]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:25.404 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:25 smithi046 ceph-mon[84949]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-06-23T01:06:25.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:25 smithi131 ceph-mon[86495]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-06-23T01:06:25.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:25 smithi092 ceph-mon[85670]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-06-23T01:06:25.685 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-06-23T01:06:25.686 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-06-23T01:06:25.686 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2023-06-23T01:06:25.686 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2023-06-23T01:06:25.686 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112165 s, 935 MB/s 2023-06-23T01:06:25.686 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2023-06-23T01:06:26.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi092:vg_nvme/lv_1 2023-06-23T01:06:26.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:26 smithi092 ceph-mon[85670]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-06-23T01:06:26.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:26 smithi092 ceph-mon[85670]: osdmap e62: 10 total, 9 up, 10 in 2023-06-23T01:06:26.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:26 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:26.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:26 smithi092 ceph-mon[85670]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:06:26.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:26 smithi092 ceph-mon[85670]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:26.421 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:06:26 smithi046 conmon[123172]: 2023-06-23T01:06:26.152+0000 7f60a4261700 -1 osd.9 0 waiting for initial osdmap 2023-06-23T01:06:26.421 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:06:26 smithi046 conmon[123172]: 2023-06-23T01:06:26.166+0000 7f609f05c700 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:06:26.421 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:26 smithi046 ceph-mon[84949]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-06-23T01:06:26.422 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:26 smithi046 ceph-mon[84949]: osdmap e62: 10 total, 9 up, 10 in 2023-06-23T01:06:26.422 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:26 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:26.422 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:26 smithi046 ceph-mon[84949]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:06:26.422 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:26 smithi046 ceph-mon[84949]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:26.481 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:26 smithi131 ceph-mon[86495]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-06-23T01:06:26.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:26 smithi131 ceph-mon[86495]: osdmap e62: 10 total, 9 up, 10 in 2023-06-23T01:06:26.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:26 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:26.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:26 smithi131 ceph-mon[86495]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2023-06-23T01:06:26.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:26 smithi131 ceph-mon[86495]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:26.636 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.b/config 2023-06-23T01:06:27.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:06:27.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: osdmap e63: 10 total, 9 up, 10 in 2023-06-23T01:06:27.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:27.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:27.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi046 2023-06-23T01:06:27.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi046 to 4293M 2023-06-23T01:06:27.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:27.484 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:27.484 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.484 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:27 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:27.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:06:27.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: osdmap e63: 10 total, 9 up, 10 in 2023-06-23T01:06:27.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:27.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:27.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi046 2023-06-23T01:06:27.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi046 to 4293M 2023-06-23T01:06:27.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:27.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:27.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.570 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:27 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:27.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2023-06-23T01:06:27.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: osdmap e63: 10 total, 9 up, 10 in 2023-06-23T01:06:27.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:27.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:27.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi046 2023-06-23T01:06:27.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:27.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi046 to 4293M 2023-06-23T01:06:27.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.642 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:27.642 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:27.642 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:27.642 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:27 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:27.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 -- 172.21.15.92:0/2888860350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f300fdee0 msgr2=0x7f4f300fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:27.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 --2- 172.21.15.92:0/2888860350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f300fdee0 0x7f4f300fe350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4f200049c0 tx=0x7f4f20009e30 comp rx=0 tx=0).stop 2023-06-23T01:06:27.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 -- 172.21.15.92:0/2888860350 shutdown_connections 2023-06-23T01:06:27.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 --2- 172.21.15.92:0/2888860350 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4f300fe890 0x7f4f30105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:27.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 --2- 172.21.15.92:0/2888860350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f300fcc30 0x7f4f300fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:27.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 --2- 172.21.15.92:0/2888860350 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f300fdee0 0x7f4f300fe350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:27.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 -- 172.21.15.92:0/2888860350 >> 172.21.15.92:0/2888860350 conn(0x7f4f300f83b0 msgr2=0x7f4f300fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:06:27.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 -- 172.21.15.92:0/2888860350 shutdown_connections 2023-06-23T01:06:27.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 -- 172.21.15.92:0/2888860350 wait complete. 2023-06-23T01:06:27.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 Processor -- start 2023-06-23T01:06:27.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 -- start start 2023-06-23T01:06:27.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f300fcc30 0x7f4f3010d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:27.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f300fdee0 0x7f4f3010dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:27.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4f300fe890 0x7f4f30114d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:27.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f30115310 con 0x7f4f300fcc30 2023-06-23T01:06:27.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f30115480 con 0x7f4f300fdee0 2023-06-23T01:06:27.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f3739e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f30115640 con 0x7f4f300fe890 2023-06-23T01:06:27.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f34939700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f300fdee0 0x7f4f3010dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:06:27.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f34939700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f300fdee0 0x7f4f3010dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:59020/0 (socket says 172.21.15.92:59020) 2023-06-23T01:06:27.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.858+0000 7f4f34939700 1 -- 172.21.15.92:0/2633991587 learned_addr learned my addr 172.21.15.92:0/2633991587 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:06:27.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f3513a700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f300fcc30 0x7f4f3010d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:06:27.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f3593b700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4f300fe890 0x7f4f30114d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:06:27.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f34939700 1 -- 172.21.15.92:0/2633991587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4f300fe890 msgr2=0x7f4f30114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:27.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f34939700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4f300fe890 0x7f4f30114d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:27.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f34939700 1 -- 172.21.15.92:0/2633991587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f300fcc30 msgr2=0x7f4f3010d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:27.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f34939700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f300fcc30 0x7f4f3010d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:27.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f34939700 1 -- 172.21.15.92:0/2633991587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f20005040 con 0x7f4f300fdee0 2023-06-23T01:06:27.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f34939700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f300fdee0 0x7f4f3010dc70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000b8f0 tx=0x7f4f20009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:06:27.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f25ffb700 1 -- 172.21.15.92:0/2633991587 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2000e040 con 0x7f4f300fdee0 2023-06-23T01:06:27.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f25ffb700 1 -- 172.21.15.92:0/2633991587 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4f20009640 con 0x7f4f300fdee0 2023-06-23T01:06:27.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f25ffb700 1 -- 172.21.15.92:0/2633991587 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f20013d80 con 0x7f4f300fdee0 2023-06-23T01:06:27.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f3739e700 1 -- 172.21.15.92:0/2633991587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f301158c0 con 0x7f4f300fdee0 2023-06-23T01:06:27.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.859+0000 7f4f3739e700 1 -- 172.21.15.92:0/2633991587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f30115cd0 con 0x7f4f300fdee0 2023-06-23T01:06:27.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.860+0000 7f4f3739e700 1 -- 172.21.15.92:0/2633991587 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f301082a0 con 0x7f4f300fdee0 2023-06-23T01:06:27.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.861+0000 7f4f25ffb700 1 -- 172.21.15.92:0/2633991587 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f4f20013420 con 0x7f4f300fdee0 2023-06-23T01:06:27.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.862+0000 7f4f25ffb700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4f1c09b780 0x7f4f1c09dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:27.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.862+0000 7f4f25ffb700 1 -- 172.21.15.92:0/2633991587 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6257+0+0 (secure 0 0 0) 0x7f4f200c5980 con 0x7f4f300fdee0 2023-06-23T01:06:27.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.862+0000 7f4f3513a700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4f1c09b780 0x7f4f1c09dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:06:27.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.863+0000 7f4f3513a700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4f1c09b780 0x7f4f1c09dc30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c004f40 tx=0x7f4f2c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:06:27.876 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:27.866+0000 7f4f25ffb700 1 -- 172.21.15.92:0/2633991587 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f4f20091940 con 0x7f4f300fdee0 2023-06-23T01:06:28.062 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:28.058+0000 7f4f3739e700 1 -- 172.21.15.92:0/2633991587 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f3005de70 con 0x7f4f1c09b780 2023-06-23T01:06:28.481 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:06:28.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:06:28.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676] boot 2023-06-23T01:06:28.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: osdmap e64: 10 total, 10 up, 10 in 2023-06-23T01:06:28.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:28.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:28.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:28.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:28.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:28.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:06:28.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:06:28.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:28.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:06:28.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:06:28.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676] boot 2023-06-23T01:06:28.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: osdmap e64: 10 total, 10 up, 10 in 2023-06-23T01:06:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:06:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:06:28.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:28.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:06:28.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:06:28.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: osd.9 [v2:172.21.15.46:6826/3042170676,v1:172.21.15.46:6827/3042170676] boot 2023-06-23T01:06:28.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: osdmap e64: 10 total, 10 up, 10 in 2023-06-23T01:06:28.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-06-23T01:06:28.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:28.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:28.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:28.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2023-06-23T01:06:28.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:06:28.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:06:28.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:29.481 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:29 smithi131 ceph-mon[86495]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:06:29.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:29 smithi131 ceph-mon[86495]: osdmap e65: 10 total, 10 up, 10 in 2023-06-23T01:06:29.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:29 smithi046 ceph-mon[84949]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:06:29.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:29 smithi046 ceph-mon[84949]: osdmap e65: 10 total, 10 up, 10 in 2023-06-23T01:06:29.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:29 smithi092 ceph-mon[85670]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:06:29.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:29 smithi092 ceph-mon[85670]: osdmap e65: 10 total, 10 up, 10 in 2023-06-23T01:06:30.481 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:30 smithi131 ceph-mon[86495]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:30.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:30 smithi046 ceph-mon[84949]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:30.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:30 smithi092 ceph-mon[85670]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:31.481 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:31 smithi131 ceph-mon[86495]: from='client.? 172.21.15.92:0/3189374187' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b4f58f5-d971-4634-a1b6-896501d3d218"}]: dispatch 2023-06-23T01:06:31.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:31 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b4f58f5-d971-4634-a1b6-896501d3d218"}]: dispatch 2023-06-23T01:06:31.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:31 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b4f58f5-d971-4634-a1b6-896501d3d218"}]': finished 2023-06-23T01:06:31.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:31 smithi131 ceph-mon[86495]: osdmap e66: 11 total, 10 up, 11 in 2023-06-23T01:06:31.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:31 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:31.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:31 smithi131 ceph-mon[86495]: from='client.? 172.21.15.92:0/2249995511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:06:31.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:31 smithi046 ceph-mon[84949]: from='client.? 172.21.15.92:0/3189374187' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b4f58f5-d971-4634-a1b6-896501d3d218"}]: dispatch 2023-06-23T01:06:31.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:31 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b4f58f5-d971-4634-a1b6-896501d3d218"}]: dispatch 2023-06-23T01:06:31.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:31 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b4f58f5-d971-4634-a1b6-896501d3d218"}]': finished 2023-06-23T01:06:31.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:31 smithi046 ceph-mon[84949]: osdmap e66: 11 total, 10 up, 11 in 2023-06-23T01:06:31.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:31 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:31.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:31 smithi046 ceph-mon[84949]: from='client.? 172.21.15.92:0/2249995511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:06:31.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:31 smithi092 ceph-mon[85670]: from='client.? 172.21.15.92:0/3189374187' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b4f58f5-d971-4634-a1b6-896501d3d218"}]: dispatch 2023-06-23T01:06:31.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:31 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b4f58f5-d971-4634-a1b6-896501d3d218"}]: dispatch 2023-06-23T01:06:31.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:31 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b4f58f5-d971-4634-a1b6-896501d3d218"}]': finished 2023-06-23T01:06:31.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:31 smithi092 ceph-mon[85670]: osdmap e66: 11 total, 10 up, 11 in 2023-06-23T01:06:31.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:31 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:31.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:31 smithi092 ceph-mon[85670]: from='client.? 172.21.15.92:0/2249995511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:06:32.481 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:32 smithi131 ceph-mon[86495]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:32.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:32 smithi046 ceph-mon[84949]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:32.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:32 smithi092 ceph-mon[85670]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:35.215 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:34 smithi092 ceph-mon[85670]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:35.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:34 smithi131 ceph-mon[86495]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:35.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:34 smithi046 ceph-mon[84949]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:37.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:36 smithi092 ceph-mon[85670]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:37.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-06-23T01:06:37.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:37.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:36 smithi131 ceph-mon[86495]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:37.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-06-23T01:06:37.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:37.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:36 smithi046 ceph-mon[84949]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:37.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-06-23T01:06:37.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:38.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:37 smithi092 ceph-mon[85670]: Deploying daemon osd.10 on smithi092 2023-06-23T01:06:38.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:37 smithi131 ceph-mon[86495]: Deploying daemon osd.10 on smithi092 2023-06-23T01:06:38.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:37 smithi046 ceph-mon[84949]: Deploying daemon osd.10 on smithi092 2023-06-23T01:06:39.017 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:38 smithi092 ceph-mon[85670]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:39.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:38 smithi131 ceph-mon[86495]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:39.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:38 smithi046 ceph-mon[84949]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:41.187 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:40 smithi092 ceph-mon[85670]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:41.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:40 smithi131 ceph-mon[86495]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:41.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:40 smithi046 ceph-mon[84949]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:42 smithi092 ceph-mon[85670]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:42.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:42.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:42.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:42.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:42 smithi046 ceph-mon[84949]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:42.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:42.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:42.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:42.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:42 smithi131 ceph-mon[86495]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:44.071 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.068+0000 7f4f25ffb700 1 -- 172.21.15.92:0/2633991587 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f4f3005de70 con 0x7f4f1c09b780 2023-06-23T01:06:44.072 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 10 on host 'smithi092' 2023-06-23T01:06:44.073 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 -- 172.21.15.92:0/2633991587 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4f1c09b780 msgr2=0x7f4f1c09dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:44.074 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4f1c09b780 0x7f4f1c09dc30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c004f40 tx=0x7f4f2c00b040 comp rx=0 tx=0).stop 2023-06-23T01:06:44.074 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 -- 172.21.15.92:0/2633991587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f300fdee0 msgr2=0x7f4f3010dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:44.074 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f300fdee0 0x7f4f3010dc70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000b8f0 tx=0x7f4f20009e30 comp rx=0 tx=0).stop 2023-06-23T01:06:44.074 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 -- 172.21.15.92:0/2633991587 shutdown_connections 2023-06-23T01:06:44.074 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f4f300fe890 0x7f4f30114d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:44.074 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f300fcc30 0x7f4f3010d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:44.075 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f4f1c09b780 0x7f4f1c09dc30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:44.075 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 --2- 172.21.15.92:0/2633991587 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4f300fdee0 0x7f4f3010dc70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:44.075 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 -- 172.21.15.92:0/2633991587 >> 172.21.15.92:0/2633991587 conn(0x7f4f300f83b0 msgr2=0x7f4f300fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:06:44.075 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 -- 172.21.15.92:0/2633991587 shutdown_connections 2023-06-23T01:06:44.075 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:06:44.070+0000 7f4f3739e700 1 -- 172.21.15.92:0/2633991587 wait complete. 2023-06-23T01:06:44.510 DEBUG:teuthology.orchestra.run.smithi092:osd.10> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.10.service 2023-06-23T01:06:44.513 INFO:tasks.cephadm:Deploying osd.11 on smithi131 with /dev/vg_nvme/lv_1... 2023-06-23T01:06:44.514 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-06-23T01:06:44.542 INFO:journalctl@ceph.osd.10.smithi092.stdout:-- Logs begin at Fri 2023-06-23 00:44:08 UTC. -- 2023-06-23T01:06:44.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:44.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:44.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:44.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:44.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:44.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:44 smithi131 ceph-mon[86495]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:44.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:44.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:44.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:45.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:45.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:45.029 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:45.030 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:45.030 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:45.030 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:44 smithi092 ceph-mon[85670]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:45.030 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:45.030 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:45.031 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:45.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:45.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:45.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:45.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:45.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:45.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:44 smithi046 ceph-mon[84949]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:45.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:45.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:45.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:06:45.104 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:06:45.338 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:06:45 smithi092 conmon[113172]: 2023-06-23T01:06:45.212+0000 7fea7af82540 -1 osd.10 0 log_to_monitors true 2023-06-23T01:06:45.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:45 smithi131 ceph-mon[86495]: from='osd.10 [v2:172.21.15.92:6824/2375910823,v1:172.21.15.92:6825/2375910823]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-23T01:06:45.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:45 smithi131 ceph-mon[86495]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-23T01:06:46.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:45 smithi046 ceph-mon[84949]: from='osd.10 [v2:172.21.15.92:6824/2375910823,v1:172.21.15.92:6825/2375910823]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-23T01:06:46.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:45 smithi046 ceph-mon[84949]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-23T01:06:46.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:45 smithi092 ceph-mon[85670]: from='osd.10 [v2:172.21.15.92:6824/2375910823,v1:172.21.15.92:6825/2375910823]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-23T01:06:46.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:45 smithi092 ceph-mon[85670]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-06-23T01:06:46.484 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-06-23T01:06:46.484 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-06-23T01:06:46.484 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2023-06-23T01:06:46.485 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2023-06-23T01:06:46.485 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110987 s, 945 MB/s 2023-06-23T01:06:46.485 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2023-06-23T01:06:46.851 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch daemon add osd smithi131:vg_nvme/lv_1 2023-06-23T01:06:46.889 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:06:46 smithi092 conmon[113172]: 2023-06-23T01:06:46.807+0000 7fea72f6a700 -1 osd.10 0 waiting for initial osdmap 2023-06-23T01:06:46.890 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:06:46 smithi092 conmon[113172]: 2023-06-23T01:06:46.822+0000 7fea6ad5f700 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:06:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-06-23T01:06:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: osdmap e67: 11 total, 10 up, 11 in 2023-06-23T01:06:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='osd.10 [v2:172.21.15.92:6824/2375910823,v1:172.21.15.92:6825/2375910823]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:06:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:06:46.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:46.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:46.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:46.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:46.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:46.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:46.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:46.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:46.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:46.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:46.892 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:47.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-06-23T01:06:47.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: osdmap e67: 11 total, 10 up, 11 in 2023-06-23T01:06:47.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:47.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='osd.10 [v2:172.21.15.92:6824/2375910823,v1:172.21.15.92:6825/2375910823]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:06:47.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:06:47.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:47.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:47.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:47.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:47.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:47.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:47.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:47.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:47.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:47.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:47.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-06-23T01:06:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: osdmap e67: 11 total, 10 up, 11 in 2023-06-23T01:06:47.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='osd.10 [v2:172.21.15.92:6824/2375910823,v1:172.21.15.92:6825/2375910823]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:06:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2023-06-23T01:06:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-06-23T01:06:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:47.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:06:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:06:47.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:06:47.433 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:06:47.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:47 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi092 2023-06-23T01:06:47.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:47 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi092 to 4292M 2023-06-23T01:06:47.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:47 smithi131 ceph-mon[86495]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:06:47.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:47 smithi131 ceph-mon[86495]: osdmap e68: 11 total, 10 up, 11 in 2023-06-23T01:06:47.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:47 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:47.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:47 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:47.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:47 smithi131 ceph-mon[86495]: osd.10 [v2:172.21.15.92:6824/2375910823,v1:172.21.15.92:6825/2375910823] boot 2023-06-23T01:06:47.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:47 smithi131 ceph-mon[86495]: osdmap e69: 11 total, 11 up, 11 in 2023-06-23T01:06:47.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:47 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:48.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:47 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi092 2023-06-23T01:06:48.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:47 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi092 to 4292M 2023-06-23T01:06:48.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:47 smithi046 ceph-mon[84949]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:06:48.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:47 smithi046 ceph-mon[84949]: osdmap e68: 11 total, 10 up, 11 in 2023-06-23T01:06:48.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:47 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:48.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:47 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:48.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:47 smithi046 ceph-mon[84949]: osd.10 [v2:172.21.15.92:6824/2375910823,v1:172.21.15.92:6825/2375910823] boot 2023-06-23T01:06:48.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:47 smithi046 ceph-mon[84949]: osdmap e69: 11 total, 11 up, 11 in 2023-06-23T01:06:48.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:47 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:48.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:47 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi092 2023-06-23T01:06:48.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:47 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi092 to 4292M 2023-06-23T01:06:48.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:47 smithi092 ceph-mon[85670]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2023-06-23T01:06:48.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:47 smithi092 ceph-mon[85670]: osdmap e68: 11 total, 10 up, 11 in 2023-06-23T01:06:48.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:47 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:48.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:47 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:48.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:47 smithi092 ceph-mon[85670]: osd.10 [v2:172.21.15.92:6824/2375910823,v1:172.21.15.92:6825/2375910823] boot 2023-06-23T01:06:48.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:47 smithi092 ceph-mon[85670]: osdmap e69: 11 total, 11 up, 11 in 2023-06-23T01:06:48.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:47 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-06-23T01:06:48.595 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.590+0000 7fa55f27d700 1 -- 172.21.15.131:0/967625272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5580fde50 msgr2=0x7fa5580fe2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:48.595 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.590+0000 7fa55f27d700 1 --2- 172.21.15.131:0/967625272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5580fde50 0x7fa5580fe2a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa5480049c0 tx=0x7fa548009e30 comp rx=0 tx=0).stop 2023-06-23T01:06:48.595 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 -- 172.21.15.131:0/967625272 shutdown_connections 2023-06-23T01:06:48.595 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 --2- 172.21.15.131:0/967625272 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5580fe7e0 0x7fa5581050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:48.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 --2- 172.21.15.131:0/967625272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5580fcc40 0x7fa5580fd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:48.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 --2- 172.21.15.131:0/967625272 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5580fde50 0x7fa5580fe2a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:48.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 -- 172.21.15.131:0/967625272 >> 172.21.15.131:0/967625272 conn(0x7fa5580f83a0 msgr2=0x7fa5580fa7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:06:48.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 -- 172.21.15.131:0/967625272 shutdown_connections 2023-06-23T01:06:48.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 -- 172.21.15.131:0/967625272 wait complete. 2023-06-23T01:06:48.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 Processor -- start 2023-06-23T01:06:48.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 -- start start 2023-06-23T01:06:48.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5580fcc40 0x7fa55810d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:48.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5580fde50 0x7fa55810dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:48.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5580fe7e0 0x7fa558114d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:48.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa558115340 con 0x7fa5580fe7e0 2023-06-23T01:06:48.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5581154b0 con 0x7fa5580fde50 2023-06-23T01:06:48.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55f27d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa558115670 con 0x7fa5580fcc40 2023-06-23T01:06:48.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55d019700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5580fcc40 0x7fa55810d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:06:48.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55d019700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5580fcc40 0x7fa55810d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:59356/0 (socket says 172.21.15.131:59356) 2023-06-23T01:06:48.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.591+0000 7fa55d019700 1 -- 172.21.15.131:0/4037236468 learned_addr learned my addr 172.21.15.131:0/4037236468 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-06-23T01:06:48.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.592+0000 7fa55d019700 1 -- 172.21.15.131:0/4037236468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5580fde50 msgr2=0x7fa55810dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:48.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.592+0000 7fa55d019700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5580fde50 0x7fa55810dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:48.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.592+0000 7fa55d019700 1 -- 172.21.15.131:0/4037236468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5580fe7e0 msgr2=0x7fa558114d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:06:48.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.592+0000 7fa55d019700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5580fe7e0 0x7fa558114d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:06:48.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.592+0000 7fa55d019700 1 -- 172.21.15.131:0/4037236468 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa548005040 con 0x7fa5580fcc40 2023-06-23T01:06:48.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.592+0000 7fa55d019700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5580fcc40 0x7fa55810d760 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa554002b00 tx=0x7fa554004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:06:48.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.592+0000 7fa54dffb700 1 -- 172.21.15.131:0/4037236468 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa554007cc0 con 0x7fa5580fcc40 2023-06-23T01:06:48.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.592+0000 7fa55f27d700 1 -- 172.21.15.131:0/4037236468 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa558115950 con 0x7fa5580fcc40 2023-06-23T01:06:48.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.592+0000 7fa54dffb700 1 -- 172.21.15.131:0/4037236468 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa55400f040 con 0x7fa5580fcc40 2023-06-23T01:06:48.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.592+0000 7fa54dffb700 1 -- 172.21.15.131:0/4037236468 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa554009340 con 0x7fa5580fcc40 2023-06-23T01:06:48.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.592+0000 7fa55f27d700 1 -- 172.21.15.131:0/4037236468 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa558115f10 con 0x7fa5580fcc40 2023-06-23T01:06:48.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.593+0000 7fa55f27d700 1 -- 172.21.15.131:0/4037236468 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa558108290 con 0x7fa5580fcc40 2023-06-23T01:06:48.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.594+0000 7fa54dffb700 1 -- 172.21.15.131:0/4037236468 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fa554009770 con 0x7fa5580fcc40 2023-06-23T01:06:48.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.595+0000 7fa54dffb700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fa54409b680 0x7fa54409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:06:48.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.595+0000 7fa54dffb700 1 -- 172.21.15.131:0/4037236468 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fa5540c4700 con 0x7fa5580fcc40 2023-06-23T01:06:48.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.595+0000 7fa55c818700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fa54409b680 0x7fa54409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:06:48.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.596+0000 7fa55c818700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fa54409b680 0x7fa54409db30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa548007f50 tx=0x7fa548009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:06:48.602 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.599+0000 7fa54dffb700 1 -- 172.21.15.131:0/4037236468 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fa5540905a0 con 0x7fa5580fcc40 2023-06-23T01:06:48.795 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:06:48.792+0000 7fa55f27d700 1 -- 172.21.15.131:0/4037236468 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa55805de70 con 0x7fa54409b680 2023-06-23T01:06:49.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:48 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:06:49.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:48 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:06:49.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:48 smithi046 ceph-mon[84949]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:06:49.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:48 smithi046 ceph-mon[84949]: osdmap e70: 11 total, 11 up, 11 in 2023-06-23T01:06:49.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:48 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:06:49.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:48 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:06:49.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:48 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:49.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:48 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:06:49.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:48 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:06:49.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:48 smithi092 ceph-mon[85670]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:06:49.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:48 smithi092 ceph-mon[85670]: osdmap e70: 11 total, 11 up, 11 in 2023-06-23T01:06:49.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:48 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:06:49.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:48 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:06:49.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:48 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:49.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:48 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:06:49.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:48 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:06:49.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:48 smithi131 ceph-mon[86495]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:06:49.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:48 smithi131 ceph-mon[86495]: osdmap e70: 11 total, 11 up, 11 in 2023-06-23T01:06:49.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:48 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-06-23T01:06:49.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:48 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-06-23T01:06:49.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:48 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:49.953 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:49 smithi131 ceph-mon[86495]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:06:49.953 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:49 smithi131 ceph-mon[86495]: osdmap e71: 11 total, 11 up, 11 in 2023-06-23T01:06:50.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:49 smithi046 ceph-mon[84949]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:06:50.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:49 smithi046 ceph-mon[84949]: osdmap e71: 11 total, 11 up, 11 in 2023-06-23T01:06:50.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:49 smithi092 ceph-mon[85670]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:06:50.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:49 smithi092 ceph-mon[85670]: osdmap e71: 11 total, 11 up, 11 in 2023-06-23T01:06:51.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:50 smithi046 ceph-mon[84949]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:06:51.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:50 smithi046 ceph-mon[84949]: osdmap e72: 11 total, 11 up, 11 in 2023-06-23T01:06:51.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:50 smithi092 ceph-mon[85670]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:06:51.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:50 smithi092 ceph-mon[85670]: osdmap e72: 11 total, 11 up, 11 in 2023-06-23T01:06:51.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:50 smithi131 ceph-mon[86495]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:06:51.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:50 smithi131 ceph-mon[86495]: osdmap e72: 11 total, 11 up, 11 in 2023-06-23T01:06:52.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:51 smithi092 ceph-mon[85670]: from='client.? 172.21.15.131:0/1109496737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d7a4115-c6e9-4cef-865b-05f57fbcface"}]: dispatch 2023-06-23T01:06:52.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:51 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d7a4115-c6e9-4cef-865b-05f57fbcface"}]: dispatch 2023-06-23T01:06:52.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:51 smithi092 ceph-mon[85670]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d7a4115-c6e9-4cef-865b-05f57fbcface"}]': finished 2023-06-23T01:06:52.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:51 smithi092 ceph-mon[85670]: osdmap e73: 12 total, 11 up, 12 in 2023-06-23T01:06:52.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:51 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:06:52.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:51 smithi092 ceph-mon[85670]: from='client.? 172.21.15.131:0/4207264261' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:06:52.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:51 smithi131 ceph-mon[86495]: from='client.? 172.21.15.131:0/1109496737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d7a4115-c6e9-4cef-865b-05f57fbcface"}]: dispatch 2023-06-23T01:06:52.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:51 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d7a4115-c6e9-4cef-865b-05f57fbcface"}]: dispatch 2023-06-23T01:06:52.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:51 smithi131 ceph-mon[86495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d7a4115-c6e9-4cef-865b-05f57fbcface"}]': finished 2023-06-23T01:06:52.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:51 smithi131 ceph-mon[86495]: osdmap e73: 12 total, 11 up, 12 in 2023-06-23T01:06:52.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:51 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:06:52.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:51 smithi131 ceph-mon[86495]: from='client.? 172.21.15.131:0/4207264261' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:06:52.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:51 smithi046 ceph-mon[84949]: from='client.? 172.21.15.131:0/1109496737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d7a4115-c6e9-4cef-865b-05f57fbcface"}]: dispatch 2023-06-23T01:06:52.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:51 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d7a4115-c6e9-4cef-865b-05f57fbcface"}]: dispatch 2023-06-23T01:06:52.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:51 smithi046 ceph-mon[84949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d7a4115-c6e9-4cef-865b-05f57fbcface"}]': finished 2023-06-23T01:06:52.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:51 smithi046 ceph-mon[84949]: osdmap e73: 12 total, 11 up, 12 in 2023-06-23T01:06:52.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:51 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:06:52.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:51 smithi046 ceph-mon[84949]: from='client.? 172.21.15.131:0/4207264261' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-06-23T01:06:53.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:52 smithi092 ceph-mon[85670]: pgmap v211: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-06-23T01:06:53.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:52 smithi131 ceph-mon[86495]: pgmap v211: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-06-23T01:06:53.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:52 smithi046 ceph-mon[84949]: pgmap v211: 1 pgs: 1 peering; 577 KiB data, 65 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-06-23T01:06:55.224 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:54 smithi131 ceph-mon[86495]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-06-23T01:06:55.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:54 smithi046 ceph-mon[84949]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-06-23T01:06:55.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:54 smithi092 ceph-mon[85670]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2023-06-23T01:06:57.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:56 smithi131 ceph-mon[86495]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-06-23T01:06:57.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:56 smithi046 ceph-mon[84949]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-06-23T01:06:57.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:56 smithi092 ceph-mon[85670]: pgmap v213: 1 pgs: 1 peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-06-23T01:06:58.481 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-06-23T01:06:58.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:58 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:58.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:58 smithi131 ceph-mon[86495]: Deploying daemon osd.11 on smithi131 2023-06-23T01:06:58.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:06:58 smithi131 ceph-mon[86495]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:06:58.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-06-23T01:06:58.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:58 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:58.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:58 smithi046 ceph-mon[84949]: Deploying daemon osd.11 on smithi131 2023-06-23T01:06:58.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:06:58 smithi046 ceph-mon[84949]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:06:58.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-06-23T01:06:58.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:58 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:06:58.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:58 smithi092 ceph-mon[85670]: Deploying daemon osd.11 on smithi131 2023-06-23T01:06:58.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:06:58 smithi092 ceph-mon[85670]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-06-23T01:07:01.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:00 smithi131 ceph-mon[86495]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-06-23T01:07:01.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:00 smithi046 ceph-mon[84949]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-06-23T01:07:01.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:00 smithi092 ceph-mon[85670]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-06-23T01:07:02.974 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:02.974 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:02.974 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:02 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:02.974 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:02 smithi131 ceph-mon[86495]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:03.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:03.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:03.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:02 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:03.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:02 smithi046 ceph-mon[84949]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:03.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:03.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:03.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:02 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:03.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:02 smithi092 ceph-mon[85670]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:04.755 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.752+0000 7fa54dffb700 1 -- 172.21.15.131:0/4037236468 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fa55805de70 con 0x7fa54409b680 2023-06-23T01:07:04.756 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 11 on host 'smithi131' 2023-06-23T01:07:04.758 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 -- 172.21.15.131:0/4037236468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fa54409b680 msgr2=0x7fa54409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:04.758 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fa54409b680 0x7fa54409db30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa548007f50 tx=0x7fa548009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:04.758 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 -- 172.21.15.131:0/4037236468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5580fcc40 msgr2=0x7fa55810d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:04.758 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5580fcc40 0x7fa55810d760 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa554002b00 tx=0x7fa554004600 comp rx=0 tx=0).stop 2023-06-23T01:07:04.759 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 -- 172.21.15.131:0/4037236468 shutdown_connections 2023-06-23T01:07:04.759 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5580fe7e0 0x7fa558114d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:04.759 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fa5580fcc40 0x7fa55810d760 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:04.759 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fa54409b680 0x7fa54409db30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:04.759 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 --2- 172.21.15.131:0/4037236468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa5580fde50 0x7fa55810dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:04.759 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 -- 172.21.15.131:0/4037236468 >> 172.21.15.131:0/4037236468 conn(0x7fa5580f83a0 msgr2=0x7fa5580fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:04.759 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 -- 172.21.15.131:0/4037236468 shutdown_connections 2023-06-23T01:07:04.760 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:04.755+0000 7fa55f27d700 1 -- 172.21.15.131:0/4037236468 wait complete. 2023-06-23T01:07:05.137 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:04 smithi131 ceph-mon[86495]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:05.138 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.138 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.138 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:05.138 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:05.138 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.139 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.139 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.139 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:04 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:05.201 DEBUG:teuthology.orchestra.run.smithi131:osd.11> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.11.service 2023-06-23T01:07:05.204 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-06-23T01:07:05.205 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph osd stat -f json 2023-06-23T01:07:05.233 INFO:journalctl@ceph.osd.11.smithi131.stdout:-- Logs begin at Fri 2023-06-23 00:44:02 UTC. -- 2023-06-23T01:07:05.237 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:04 smithi046 ceph-mon[84949]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:05.237 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.237 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.237 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:05.237 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:05.238 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.238 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.238 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.238 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:04 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:05.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:04 smithi092 ceph-mon[85670]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:05.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:05.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:05.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:05.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:04 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:05.816 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:07:05.904 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:07:05 smithi131 conmon[115245]: 2023-06-23T01:07:05.746+0000 7f07de1e8540 -1 osd.11 0 log_to_monitors true 2023-06-23T01:07:06.193 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:05 smithi046 ceph-mon[84949]: from='osd.11 [v2:172.21.15.131:6824/1716414800,v1:172.21.15.131:6825/1716414800]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-23T01:07:06.193 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:05 smithi046 ceph-mon[84949]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-23T01:07:06.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:05 smithi131 ceph-mon[86495]: from='osd.11 [v2:172.21.15.131:6824/1716414800,v1:172.21.15.131:6825/1716414800]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-23T01:07:06.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:05 smithi131 ceph-mon[86495]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-23T01:07:06.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:05 smithi092 ceph-mon[85670]: from='osd.11 [v2:172.21.15.131:6824/1716414800,v1:172.21.15.131:6825/1716414800]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-23T01:07:06.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:05 smithi092 ceph-mon[85670]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-06-23T01:07:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.073+0000 7fcef1162700 1 -- 172.21.15.46:0/2998336409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceec0fde60 msgr2=0x7fceec0fe2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.073+0000 7fcef1162700 1 --2- 172.21.15.46:0/2998336409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceec0fde60 0x7fceec0fe2b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0049c0 tx=0x7fcedc009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.073+0000 7fcef1162700 1 -- 172.21.15.46:0/2998336409 shutdown_connections 2023-06-23T01:07:07.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.073+0000 7fcef1162700 1 --2- 172.21.15.46:0/2998336409 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fceec0fcc50 0x7fceec0fd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.073+0000 7fcef1162700 1 --2- 172.21.15.46:0/2998336409 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fceec0fe7f0 0x7fceec1050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.073+0000 7fcef1162700 1 --2- 172.21.15.46:0/2998336409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceec0fde60 0x7fceec0fe2b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.073+0000 7fcef1162700 1 -- 172.21.15.46:0/2998336409 >> 172.21.15.46:0/2998336409 conn(0x7fceec0f83b0 msgr2=0x7fceec0fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.073+0000 7fcef1162700 1 -- 172.21.15.46:0/2998336409 shutdown_connections 2023-06-23T01:07:07.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.073+0000 7fcef1162700 1 -- 172.21.15.46:0/2998336409 wait complete. 2023-06-23T01:07:07.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fcef1162700 1 Processor -- start 2023-06-23T01:07:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fcef1162700 1 -- start start 2023-06-23T01:07:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fcef1162700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fceec0fcc50 0x7fceec071720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fcef1162700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fceec0fde60 0x7fceec06fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:07.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fcef1162700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceec0fe7f0 0x7fceec0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fcef1162700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec070a30 con 0x7fceec0fe7f0 2023-06-23T01:07:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fcef1162700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec070ba0 con 0x7fceec0fde60 2023-06-23T01:07:07.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fcef1162700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec1178c0 con 0x7fceec0fcc50 2023-06-23T01:07:07.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceeb59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceec0fe7f0 0x7fceec0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:07.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceeb59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceec0fe7f0 0x7fceec0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44656/0 (socket says 172.21.15.46:44656) 2023-06-23T01:07:07.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceeb59e700 1 -- 172.21.15.46:0/2227937177 learned_addr learned my addr 172.21.15.46:0/2227937177 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:07.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceead9d700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fceec0fcc50 0x7fceec071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:07.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceea59c700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fceec0fde60 0x7fceec06fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:07.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceeb59e700 1 -- 172.21.15.46:0/2227937177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fceec0fcc50 msgr2=0x7fceec071720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:07.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceeb59e700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fceec0fcc50 0x7fceec071720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:07.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceeb59e700 1 -- 172.21.15.46:0/2227937177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fceec0fde60 msgr2=0x7fceec06fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:07.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceeb59e700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fceec0fde60 0x7fceec06fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:07.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceeb59e700 1 -- 172.21.15.46:0/2227937177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc005040 con 0x7fceec0fe7f0 2023-06-23T01:07:07.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceead9d700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fceec0fcc50 0x7fceec071720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:07:07.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.074+0000 7fceeb59e700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceec0fe7f0 0x7fceec0702c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcee400ec30 tx=0x7fcee400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:07.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.075+0000 7fced37fe700 1 -- 172.21.15.46:0/2227937177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4007b30 con 0x7fceec0fe7f0 2023-06-23T01:07:07.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.075+0000 7fcef1162700 1 -- 172.21.15.46:0/2227937177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceec117b70 con 0x7fceec0fe7f0 2023-06-23T01:07:07.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.075+0000 7fced37fe700 1 -- 172.21.15.46:0/2227937177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcee400f040 con 0x7fceec0fe7f0 2023-06-23T01:07:07.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.075+0000 7fced37fe700 1 -- 172.21.15.46:0/2227937177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee400ca20 con 0x7fceec0fe7f0 2023-06-23T01:07:07.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.075+0000 7fcef1162700 1 -- 172.21.15.46:0/2227937177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceec118060 con 0x7fceec0fe7f0 2023-06-23T01:07:07.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.076+0000 7fcef1162700 1 -- 172.21.15.46:0/2227937177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceec1082f0 con 0x7fceec0fe7f0 2023-06-23T01:07:07.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.077+0000 7fced37fe700 1 -- 172.21.15.46:0/2227937177 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fcee4009090 con 0x7fceec0fe7f0 2023-06-23T01:07:07.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.077+0000 7fced37fe700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fced409b680 0x7fced409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:07.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.077+0000 7fced37fe700 1 -- 172.21.15.46:0/2227937177 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6843+0+0 (secure 0 0 0) 0x7fcee40c5380 con 0x7fceec0fe7f0 2023-06-23T01:07:07.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.077+0000 7fceead9d700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fced409b680 0x7fced409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:07.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.078+0000 7fceead9d700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fced409b680 0x7fced409db30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcee0004600 tx=0x7fcee000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:07.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.081+0000 7fced37fe700 1 -- 172.21.15.46:0/2227937177 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fcee40911a0 con 0x7fceec0fe7f0 2023-06-23T01:07:07.176 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:07:06 smithi131 conmon[115245]: 2023-06-23T01:07:06.919+0000 7f07d49cd700 -1 osd.11 0 waiting for initial osdmap 2023-06-23T01:07:07.176 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:07:06 smithi131 conmon[115245]: 2023-06-23T01:07:06.938+0000 7f07cd7c4700 -1 osd.11 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:07:07.177 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:06 smithi131 ceph-mon[86495]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:07.177 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:06 smithi131 ceph-mon[86495]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-06-23T01:07:07.177 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:06 smithi131 ceph-mon[86495]: osdmap e74: 12 total, 11 up, 12 in 2023-06-23T01:07:07.177 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:06 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:07.177 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:06 smithi131 ceph-mon[86495]: from='osd.11 [v2:172.21.15.131:6824/1716414800,v1:172.21.15.131:6825/1716414800]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:07:07.178 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:06 smithi131 ceph-mon[86495]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:07:07.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.274+0000 7fcef1162700 1 -- 172.21.15.46:0/2227937177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fceec118250 con 0x7fceec0fe7f0 2023-06-23T01:07:07.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.274+0000 7fced37fe700 1 -- 172.21.15.46:0/2227937177 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) v1 ==== 74+0+133 (secure 0 0 0) 0x7fcee40947c0 con 0x7fceec0fe7f0 2023-06-23T01:07:07.278 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 -- 172.21.15.46:0/2227937177 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fced409b680 msgr2=0x7fced409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fced409b680 0x7fced409db30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcee0004600 tx=0x7fcee000b040 comp rx=0 tx=0).stop 2023-06-23T01:07:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 -- 172.21.15.46:0/2227937177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceec0fe7f0 msgr2=0x7fceec0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceec0fe7f0 0x7fceec0702c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcee400ec30 tx=0x7fcee400b040 comp rx=0 tx=0).stop 2023-06-23T01:07:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 -- 172.21.15.46:0/2227937177 shutdown_connections 2023-06-23T01:07:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fceec0fe7f0 0x7fceec0702c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fceec0fcc50 0x7fceec071720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fced409b680 0x7fced409db30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 --2- 172.21.15.46:0/2227937177 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fceec0fde60 0x7fceec06fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 -- 172.21.15.46:0/2227937177 >> 172.21.15.46:0/2227937177 conn(0x7fceec0f83b0 msgr2=0x7fceec0fa5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 -- 172.21.15.46:0/2227937177 shutdown_connections 2023-06-23T01:07:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:07.276+0000 7fcef1162700 1 -- 172.21.15.46:0/2227937177 wait complete. 2023-06-23T01:07:07.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:06 smithi046 ceph-mon[84949]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:07.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:06 smithi046 ceph-mon[84949]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-06-23T01:07:07.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:06 smithi046 ceph-mon[84949]: osdmap e74: 12 total, 11 up, 12 in 2023-06-23T01:07:07.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:06 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:07.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:06 smithi046 ceph-mon[84949]: from='osd.11 [v2:172.21.15.131:6824/1716414800,v1:172.21.15.131:6825/1716414800]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:07:07.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:06 smithi046 ceph-mon[84949]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:07:07.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:06 smithi092 ceph-mon[85670]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:07.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:06 smithi092 ceph-mon[85670]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-06-23T01:07:07.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:06 smithi092 ceph-mon[85670]: osdmap e74: 12 total, 11 up, 12 in 2023-06-23T01:07:07.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:06 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:07.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:06 smithi092 ceph-mon[85670]: from='osd.11 [v2:172.21.15.131:6824/1716414800,v1:172.21.15.131:6825/1716414800]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:07:07.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:06 smithi092 ceph-mon[85670]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2023-06-23T01:07:07.854 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":75,"num_osds":12,"num_up_osds":11,"osd_up_since":1687482407,"num_in_osds":12,"osd_in_since":1687482411,"num_remapped_pgs":0} 2023-06-23T01:07:08.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:07:08.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: osdmap e75: 12 total, 11 up, 12 in 2023-06-23T01:07:08.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:08.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:08.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2227937177' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-23T01:07:08.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:08.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:08.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:07 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:07:08.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: osdmap e75: 12 total, 11 up, 12 in 2023-06-23T01:07:08.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:08.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:08.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2227937177' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-23T01:07:08.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:08.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:08.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:07 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2023-06-23T01:07:08.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: osdmap e75: 12 total, 11 up, 12 in 2023-06-23T01:07:08.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:08.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:08.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2227937177' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-23T01:07:08.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-06-23T01:07:08.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:08.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:08.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:07 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:08.855 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph osd stat -f json 2023-06-23T01:07:09.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:08 smithi131 ceph-mon[86495]: purged_snaps scrub starts 2023-06-23T01:07:09.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:08 smithi131 ceph-mon[86495]: purged_snaps scrub ok 2023-06-23T01:07:09.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:08 smithi131 ceph-mon[86495]: Detected new or changed devices on smithi131 2023-06-23T01:07:09.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:08 smithi131 ceph-mon[86495]: Adjusting osd_memory_target on smithi131 to 4292M 2023-06-23T01:07:09.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:08 smithi131 ceph-mon[86495]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:09.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:08 smithi131 ceph-mon[86495]: osd.11 [v2:172.21.15.131:6824/1716414800,v1:172.21.15.131:6825/1716414800] boot 2023-06-23T01:07:09.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:08 smithi131 ceph-mon[86495]: osdmap e76: 12 total, 12 up, 12 in 2023-06-23T01:07:09.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:08 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:09.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:08 smithi046 ceph-mon[84949]: purged_snaps scrub starts 2023-06-23T01:07:09.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:08 smithi046 ceph-mon[84949]: purged_snaps scrub ok 2023-06-23T01:07:09.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:08 smithi046 ceph-mon[84949]: Detected new or changed devices on smithi131 2023-06-23T01:07:09.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:08 smithi046 ceph-mon[84949]: Adjusting osd_memory_target on smithi131 to 4292M 2023-06-23T01:07:09.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:08 smithi046 ceph-mon[84949]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:09.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:08 smithi046 ceph-mon[84949]: osd.11 [v2:172.21.15.131:6824/1716414800,v1:172.21.15.131:6825/1716414800] boot 2023-06-23T01:07:09.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:08 smithi046 ceph-mon[84949]: osdmap e76: 12 total, 12 up, 12 in 2023-06-23T01:07:09.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:08 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:09.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:08 smithi092 ceph-mon[85670]: purged_snaps scrub starts 2023-06-23T01:07:09.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:08 smithi092 ceph-mon[85670]: purged_snaps scrub ok 2023-06-23T01:07:09.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:08 smithi092 ceph-mon[85670]: Detected new or changed devices on smithi131 2023-06-23T01:07:09.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:08 smithi092 ceph-mon[85670]: Adjusting osd_memory_target on smithi131 to 4292M 2023-06-23T01:07:09.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:08 smithi092 ceph-mon[85670]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-06-23T01:07:09.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:08 smithi092 ceph-mon[85670]: osd.11 [v2:172.21.15.131:6824/1716414800,v1:172.21.15.131:6825/1716414800] boot 2023-06-23T01:07:09.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:08 smithi092 ceph-mon[85670]: osdmap e76: 12 total, 12 up, 12 in 2023-06-23T01:07:09.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:08 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-06-23T01:07:09.462 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:07:10.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:10 smithi131 ceph-mon[86495]: osdmap e77: 12 total, 12 up, 12 in 2023-06-23T01:07:10.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:10 smithi131 ceph-mon[86495]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:10.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:10 smithi046 ceph-mon[84949]: osdmap e77: 12 total, 12 up, 12 in 2023-06-23T01:07:10.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:10 smithi046 ceph-mon[84949]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.877+0000 7fb856bef700 1 -- 172.21.15.46:0/2745960309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8500fe930 msgr2=0x7fb8501051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.877+0000 7fb856bef700 1 --2- 172.21.15.46:0/2745960309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8500fe930 0x7fb8501051f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb838004a20 tx=0x7fb838009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.877+0000 7fb856bef700 1 -- 172.21.15.46:0/2745960309 shutdown_connections 2023-06-23T01:07:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.877+0000 7fb856bef700 1 --2- 172.21.15.46:0/2745960309 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8500faf10 0x7fb8500fb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.877+0000 7fb856bef700 1 --2- 172.21.15.46:0/2745960309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8500fe930 0x7fb8501051f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.877+0000 7fb856bef700 1 --2- 172.21.15.46:0/2745960309 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8500fdfa0 0x7fb8500fe3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.877+0000 7fb856bef700 1 -- 172.21.15.46:0/2745960309 >> 172.21.15.46:0/2745960309 conn(0x7fb8500f6a50 msgr2=0x7fb8500f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.877+0000 7fb856bef700 1 -- 172.21.15.46:0/2745960309 shutdown_connections 2023-06-23T01:07:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.877+0000 7fb856bef700 1 -- 172.21.15.46:0/2745960309 wait complete. 2023-06-23T01:07:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb856bef700 1 Processor -- start 2023-06-23T01:07:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb856bef700 1 -- start start 2023-06-23T01:07:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb856bef700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8500faf10 0x7fb85010d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb856bef700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8500fdfa0 0x7fb85010da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb856bef700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8500fe930 0x7fb850114b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb856bef700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb850115110 con 0x7fb8500faf10 2023-06-23T01:07:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb856bef700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb850115280 con 0x7fb8500fdfa0 2023-06-23T01:07:10.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb856bef700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb850115440 con 0x7fb8500fe930 2023-06-23T01:07:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb85498b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8500faf10 0x7fb85010d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb85498b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8500faf10 0x7fb85010d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44670/0 (socket says 172.21.15.46:44670) 2023-06-23T01:07:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb85498b700 1 -- 172.21.15.46:0/1610402365 learned_addr learned my addr 172.21.15.46:0/1610402365 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb84ffff700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8500fdfa0 0x7fb85010da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.878+0000 7fb85518c700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8500fe930 0x7fb850114b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.879+0000 7fb85498b700 1 -- 172.21.15.46:0/1610402365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8500fe930 msgr2=0x7fb850114b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.879+0000 7fb85498b700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8500fe930 0x7fb850114b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.879+0000 7fb85498b700 1 -- 172.21.15.46:0/1610402365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8500fdfa0 msgr2=0x7fb85010da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.879+0000 7fb85498b700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8500fdfa0 0x7fb85010da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.879+0000 7fb85498b700 1 -- 172.21.15.46:0/1610402365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb838005040 con 0x7fb8500faf10 2023-06-23T01:07:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.879+0000 7fb85498b700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8500faf10 0x7fb85010d530 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb84800ecb0 tx=0x7fb84800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.879+0000 7fb84d7fa700 1 -- 172.21.15.46:0/1610402365 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb848007b30 con 0x7fb8500faf10 2023-06-23T01:07:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.879+0000 7fb856bef700 1 -- 172.21.15.46:0/1610402365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb850115720 con 0x7fb8500faf10 2023-06-23T01:07:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.879+0000 7fb84d7fa700 1 -- 172.21.15.46:0/1610402365 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb84800f040 con 0x7fb8500faf10 2023-06-23T01:07:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.879+0000 7fb84d7fa700 1 -- 172.21.15.46:0/1610402365 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84800c980 con 0x7fb8500faf10 2023-06-23T01:07:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.879+0000 7fb856bef700 1 -- 172.21.15.46:0/1610402365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb850115c10 con 0x7fb8500faf10 2023-06-23T01:07:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.880+0000 7fb856bef700 1 -- 172.21.15.46:0/1610402365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb850108370 con 0x7fb8500faf10 2023-06-23T01:07:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.881+0000 7fb84d7fa700 1 -- 172.21.15.46:0/1610402365 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fb848009110 con 0x7fb8500faf10 2023-06-23T01:07:10.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.881+0000 7fb84d7fa700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb83c09b680 0x7fb83c09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:10.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.882+0000 7fb84ffff700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb83c09b680 0x7fb83c09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:10.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.882+0000 7fb84d7fa700 1 -- 172.21.15.46:0/1610402365 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fb8480c4bf0 con 0x7fb8500faf10 2023-06-23T01:07:10.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.882+0000 7fb84ffff700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb83c09b680 0x7fb83c09db30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb8400041d0 tx=0x7fb8400043c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:10.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:10.885+0000 7fb84d7fa700 1 -- 172.21.15.46:0/1610402365 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fb848090850 con 0x7fb8500faf10 2023-06-23T01:07:10.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:10 smithi092 ceph-mon[85670]: osdmap e77: 12 total, 12 up, 12 in 2023-06-23T01:07:10.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:10 smithi092 ceph-mon[85670]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.078+0000 7fb856bef700 1 -- 172.21.15.46:0/1610402365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fb850103940 con 0x7fb8500faf10 2023-06-23T01:07:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.079+0000 7fb84d7fa700 1 -- 172.21.15.46:0/1610402365 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v77) v1 ==== 74+0+133 (secure 0 0 0) 0x7fb8480230a0 con 0x7fb8500faf10 2023-06-23T01:07:11.083 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:11.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 -- 172.21.15.46:0/1610402365 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb83c09b680 msgr2=0x7fb83c09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:11.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb83c09b680 0x7fb83c09db30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb8400041d0 tx=0x7fb8400043c0 comp rx=0 tx=0).stop 2023-06-23T01:07:11.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 -- 172.21.15.46:0/1610402365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8500faf10 msgr2=0x7fb85010d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8500faf10 0x7fb85010d530 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb84800ecb0 tx=0x7fb84800b040 comp rx=0 tx=0).stop 2023-06-23T01:07:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 -- 172.21.15.46:0/1610402365 shutdown_connections 2023-06-23T01:07:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8500fe930 0x7fb850114b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8500faf10 0x7fb85010d530 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb83c09b680 0x7fb83c09db30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 --2- 172.21.15.46:0/1610402365 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8500fdfa0 0x7fb85010da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:11.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 -- 172.21.15.46:0/1610402365 >> 172.21.15.46:0/1610402365 conn(0x7fb8500f6a50 msgr2=0x7fb8500f8930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:11.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 -- 172.21.15.46:0/1610402365 shutdown_connections 2023-06-23T01:07:11.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:11.081+0000 7fb856bef700 1 -- 172.21.15.46:0/1610402365 wait complete. 2023-06-23T01:07:11.599 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":77,"num_osds":12,"num_up_osds":12,"osd_up_since":1687482427,"num_in_osds":12,"osd_in_since":1687482411,"num_remapped_pgs":0} 2023-06-23T01:07:11.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph osd dump --format=json 2023-06-23T01:07:11.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:11 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/1610402365' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-23T01:07:11.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:11 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/1610402365' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-23T01:07:11.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:11 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/1610402365' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-06-23T01:07:12.194 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:07:12.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:12 smithi131 ceph-mon[86495]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:12.756 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:12 smithi046 ceph-mon[84949]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:12.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:12 smithi092 ceph-mon[85670]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 72 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:13.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.594+0000 7f21a59d8700 1 -- 172.21.15.46:0/668284801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00fb8b0 msgr2=0x7f21a0109aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.594+0000 7f21a59d8700 1 --2- 172.21.15.46:0/668284801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00fb8b0 0x7f21a0109aa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2198009d60 tx=0x7f2198004d10 comp rx=0 tx=0).stop 2023-06-23T01:07:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.594+0000 7f21a59d8700 1 -- 172.21.15.46:0/668284801 shutdown_connections 2023-06-23T01:07:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.594+0000 7f21a59d8700 1 --2- 172.21.15.46:0/668284801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a00fb8b0 0x7f21a0109aa0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.594+0000 7f21a59d8700 1 --2- 172.21.15.46:0/668284801 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21a0102cc0 0x7f21a0103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.594+0000 7f21a59d8700 1 --2- 172.21.15.46:0/668284801 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21a0103660 0x7f21a00fb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.594+0000 7f21a59d8700 1 -- 172.21.15.46:0/668284801 >> 172.21.15.46:0/668284801 conn(0x7f21a00751b0 msgr2=0x7f21a00f93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.594+0000 7f21a59d8700 1 -- 172.21.15.46:0/668284801 shutdown_connections 2023-06-23T01:07:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.594+0000 7f21a59d8700 1 -- 172.21.15.46:0/668284801 wait complete. 2023-06-23T01:07:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f21a59d8700 1 Processor -- start 2023-06-23T01:07:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f21a59d8700 1 -- start start 2023-06-23T01:07:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f21a59d8700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21a00fb8b0 0x7f21a00fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f21a59d8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a0102cc0 0x7f21a00fc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f21a59d8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21a0103660 0x7f21a00fcde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f21a59d8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a011bc20 con 0x7f21a0102cc0 2023-06-23T01:07:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f21a59d8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a011bd90 con 0x7f21a0103660 2023-06-23T01:07:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f21a59d8700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a011bf50 con 0x7f21a00fb8b0 2023-06-23T01:07:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f219e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a0102cc0 0x7f21a00fc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f219e7fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a0102cc0 0x7f21a00fc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44686/0 (socket says 172.21.15.46:44686) 2023-06-23T01:07:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f219e7fc700 1 -- 172.21.15.46:0/1764362790 learned_addr learned my addr 172.21.15.46:0/1764362790 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f219effd700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21a00fb8b0 0x7f21a00fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.595+0000 7f219f7fe700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21a0103660 0x7f21a00fcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f219e7fc700 1 -- 172.21.15.46:0/1764362790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21a00fb8b0 msgr2=0x7f21a00fc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:13.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f219e7fc700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21a00fb8b0 0x7f21a00fc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:13.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f219e7fc700 1 -- 172.21.15.46:0/1764362790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21a0103660 msgr2=0x7f21a00fcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:13.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f219e7fc700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21a0103660 0x7f21a00fcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:13.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f219e7fc700 1 -- 172.21.15.46:0/1764362790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2198009940 con 0x7f21a0102cc0 2023-06-23T01:07:13.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f219f7fe700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21a0103660 0x7f21a00fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:07:13.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f219e7fc700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a0102cc0 0x7f21a00fc8a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f219400c880 tx=0x7f2194004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:13.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f21877fe700 1 -- 172.21.15.46:0/1764362790 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219400a520 con 0x7f21a0102cc0 2023-06-23T01:07:13.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f21877fe700 1 -- 172.21.15.46:0/1764362790 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f219400ab20 con 0x7f21a0102cc0 2023-06-23T01:07:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f21a59d8700 1 -- 172.21.15.46:0/1764362790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a011c230 con 0x7f21a0102cc0 2023-06-23T01:07:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f21877fe700 1 -- 172.21.15.46:0/1764362790 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194012c00 con 0x7f21a0102cc0 2023-06-23T01:07:13.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.596+0000 7f21a59d8700 1 -- 172.21.15.46:0/1764362790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a011c720 con 0x7f21a0102cc0 2023-06-23T01:07:13.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.597+0000 7f21a59d8700 1 -- 172.21.15.46:0/1764362790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a0063d10 con 0x7f21a0102cc0 2023-06-23T01:07:13.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.598+0000 7f21877fe700 1 -- 172.21.15.46:0/1764362790 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f219400a680 con 0x7f21a0102cc0 2023-06-23T01:07:13.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.598+0000 7f21877fe700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f218809b680 0x7f218809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:13.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.599+0000 7f219effd700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f218809b680 0x7f218809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:13.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.599+0000 7f21877fe700 1 -- 172.21.15.46:0/1764362790 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f21940c47e0 con 0x7f21a0102cc0 2023-06-23T01:07:13.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.599+0000 7f219effd700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f218809b680 0x7f218809db30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f21a00650e0 tx=0x7f2190009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:13.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.603+0000 7f21877fe700 1 -- 172.21.15.46:0/1764362790 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f2194090440 con 0x7f21a0102cc0 2023-06-23T01:07:13.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.800+0000 7f21a59d8700 1 -- 172.21.15.46:0/1764362790 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21a010ca10 con 0x7f21a0102cc0 2023-06-23T01:07:13.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.801+0000 7f21877fe700 1 -- 172.21.15.46:0/1764362790 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18097 (secure 0 0 0) 0x7f2194093a60 con 0x7f21a0102cc0 2023-06-23T01:07:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:13.805 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":77,"fsid":"458642ee-1161-11ee-9b2b-001a4aab830c","created":"2023-06-23T01:00:57.877538+0000","modified":"2023-06-23T01:07:09.444468+0000","last_up_change":"2023-06-23T01:07:07.914869+0000","last_in_change":"2023-06-23T01:06:51.140023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:04:00.299796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f28767ce-a780-4e50-b118-37a8c275c0e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6803","nonce":709285010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6805","nonce":709285010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6809","nonce":709285010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6807","nonce":709285010}]},"public_addr":"172.21.15.46:6803/709285010","cluster_addr":"172.21.15.46:6805/709285010","heartbeat_back_addr":"172.21.15.46:6809/709285010","heartbeat_front_addr":"172.21.15.46:6807/709285010","state":["exists","up"]},{"osd":1,"uuid":"4d4c4c36-d4a1-4b96-9059-adffee04401a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6801","nonce":4266072183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6803","nonce":4266072183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6807","nonce":4266072183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6805","nonce":4266072183}]},"public_addr":"172.21.15.92:6801/4266072183","cluster_addr":"172.21.15.92:6803/4266072183","heartbeat_back_addr":"172.21.15.92:6807/4266072183","heartbeat_front_addr":"172.21.15.92:6805/4266072183","state":["exists","up"]},{"osd":2,"uuid":"ec7b0c60-88ea-4c2b-a34c-96d188e02014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6801","nonce":1787957441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6803","nonce":1787957441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6807","nonce":1787957441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6805","nonce":1787957441}]},"public_addr":"172.21.15.131:6801/1787957441","cluster_addr":"172.21.15.131:6803/1787957441","heartbeat_back_addr":"172.21.15.131:6807/1787957441","heartbeat_front_addr":"172.21.15.131:6805/1787957441","state":["exists","up"]},{"osd":3,"uuid":"32670a23-3286-4538-9628-38d763429cbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6811","nonce":338690444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6813","nonce":338690444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6817","nonce":338690444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6815","nonce":338690444}]},"public_addr":"172.21.15.46:6811/338690444","cluster_addr":"172.21.15.46:6813/338690444","heartbeat_back_addr":"172.21.15.46:6817/338690444","heartbeat_front_addr":"172.21.15.46:6815/338690444","state":["exists","up"]},{"osd":4,"uuid":"0d8f982c-eb74-48a2-949e-d0ccd7732937","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6809","nonce":2024629902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6811","nonce":2024629902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6815","nonce":2024629902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6813","nonce":2024629902}]},"public_addr":"172.21.15.92:6809/2024629902","cluster_addr":"172.21.15.92:6811/2024629902","heartbeat_back_addr":"172.21.15.92:6815/2024629902","heartbeat_front_addr":"172.21.15.92:6813/2024629902","state":["exists","up"]},{"osd":5,"uuid":"52a7160c-4735-4a77-bc08-4d411c282b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6809","nonce":2577961971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6811","nonce":2577961971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6815","nonce":2577961971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6813","nonce":2577961971}]},"public_addr":"172.21.15.131:6809/2577961971","cluster_addr":"172.21.15.131:6811/2577961971","heartbeat_back_addr":"172.21.15.131:6815/2577961971","heartbeat_front_addr":"172.21.15.131:6813/2577961971","state":["exists","up"]},{"osd":6,"uuid":"3f3d2690-f655-4dda-a36c-bc97f9d08a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6819","nonce":558288790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6821","nonce":558288790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6825","nonce":558288790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6823","nonce":558288790}]},"public_addr":"172.21.15.46:6819/558288790","cluster_addr":"172.21.15.46:6821/558288790","heartbeat_back_addr":"172.21.15.46:6825/558288790","heartbeat_front_addr":"172.21.15.46:6823/558288790","state":["exists","up"]},{"osd":7,"uuid":"2eb43609-8921-4891-b0c4-af932bfb69bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6817","nonce":3796540277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6819","nonce":3796540277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6823","nonce":3796540277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6821","nonce":3796540277}]},"public_addr":"172.21.15.92:6817/3796540277","cluster_addr":"172.21.15.92:6819/3796540277","heartbeat_back_addr":"172.21.15.92:6823/3796540277","heartbeat_front_addr":"172.21.15.92:6821/3796540277","state":["exists","up"]},{"osd":8,"uuid":"0139b39e-f933-4685-b056-a6c27fa87fd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6817","nonce":1004740496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6819","nonce":1004740496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6823","nonce":1004740496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6821","nonce":1004740496}]},"public_addr":"172.21.15.131:6817/1004740496","cluster_addr":"172.21.15.131:6819/1004740496","heartbeat_back_addr":"172.21.15.131:6823/1004740496","heartbeat_front_addr":"172.21.15.131:6821/1004740496","state":["exists","up"]},{"osd":9,"uuid":"8bd083b4-f766-4e1b-b959-19d2fa82d77f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6827","nonce":3042170676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6829","nonce":3042170676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6833","nonce":3042170676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6831","nonce":3042170676}]},"public_addr":"172.21.15.46:6827/3042170676","cluster_addr":"172.21.15.46:6829/3042170676","heartbeat_back_addr":"172.21.15.46:6833/3042170676","heartbeat_front_addr":"172.21.15.46:6831/3042170676","state":["exists","up"]},{"osd":10,"uuid":"0b4f58f5-d971-4634-a1b6-896501d3d218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6825","nonce":2375910823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6827","nonce":2375910823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6831","nonce":2375910823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6829","nonce":2375910823}]},"public_addr":"172.21.15.92:6825/2375910823","cluster_addr":"172.21.15.92:6827/2375910823","heartbeat_back_addr":"172.21.15.92:6831/2375910823","heartbeat_front_addr":"172.21.15.92:6829/2375910823","state":["exists","up"]},{"osd":11,"uuid":"7d7a4115-c6e9-4cef-865b-05f57fbcface","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6825","nonce":1716414800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6827","nonce":1716414800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6831","nonce":1716414800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6829","nonce":1716414800}]},"public_addr":"172.21.15.131:6825/1716414800","cluster_addr":"172.21.15.131:6827/1716414800","heartbeat_back_addr":"172.21.15.131:6831/1716414800","heartbeat_front_addr":"172.21.15.131:6829/1716414800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:15.785940+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:36.479807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:57.678238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:04:20.396101+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:04:41.179881+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:01.570524+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:22.313074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:43.129445+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:03.897470+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:25.196569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:46.208577+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:07:06.788656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/685567953":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6801/2331157358":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6800/2331157358":"2023-06-24T01:01:55.786463+0000","172.21.15.46:0/1655911006":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/187730254":"2023-06-24T01:01:32.095580+0000","172.21.15.46:6801/3291477171":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/3127678959":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6800/3291477171":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/411746526":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/360533287":"2023-06-24T01:01:55.786463+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:07:13.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.803+0000 7f21a59d8700 1 -- 172.21.15.46:0/1764362790 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f218809b680 msgr2=0x7f218809db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:13.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.803+0000 7f21a59d8700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f218809b680 0x7f218809db30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f21a00650e0 tx=0x7f2190009040 comp rx=0 tx=0).stop 2023-06-23T01:07:13.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.803+0000 7f21a59d8700 1 -- 172.21.15.46:0/1764362790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a0102cc0 msgr2=0x7f21a00fc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:13.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.803+0000 7f21a59d8700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a0102cc0 0x7f21a00fc8a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f219400c880 tx=0x7f2194004470 comp rx=0 tx=0).stop 2023-06-23T01:07:13.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.804+0000 7f21a59d8700 1 -- 172.21.15.46:0/1764362790 shutdown_connections 2023-06-23T01:07:13.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.804+0000 7f21a59d8700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f218809b680 0x7f218809db30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:13.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.804+0000 7f21a59d8700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f21a0103660 0x7f21a00fcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:13.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.804+0000 7f21a59d8700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f21a00fb8b0 0x7f21a00fc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:13.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.804+0000 7f21a59d8700 1 --2- 172.21.15.46:0/1764362790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21a0102cc0 0x7f21a00fc8a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:13.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.804+0000 7f21a59d8700 1 -- 172.21.15.46:0/1764362790 >> 172.21.15.46:0/1764362790 conn(0x7f21a00751b0 msgr2=0x7f21a01008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:13.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.804+0000 7f21a59d8700 1 -- 172.21.15.46:0/1764362790 shutdown_connections 2023-06-23T01:07:13.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:13.804+0000 7f21a59d8700 1 -- 172.21.15.46:0/1764362790 wait complete. 2023-06-23T01:07:14.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:13 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/1764362790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:14.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:13 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/1764362790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:14.232 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-23T01:04:00.299796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-23T01:07:14.232 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-06-23T01:07:14.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:13 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/1764362790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.a/config 2023-06-23T01:07:15.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:14 smithi131 ceph-mon[86495]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:15.272 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:14 smithi046 ceph-mon[84949]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:15.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:14 smithi092 ceph-mon[85670]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.127+0000 7f82f4c5f700 1 -- 172.21.15.46:0/856378301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f00fce60 msgr2=0x7f82f0108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.127+0000 7f82f4c5f700 1 --2- 172.21.15.46:0/856378301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f00fce60 0x7f82f0108930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009d60 tx=0x7f82e8004d10 comp rx=0 tx=0).stop 2023-06-23T01:07:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.127+0000 7f82f4c5f700 1 -- 172.21.15.46:0/856378301 shutdown_connections 2023-06-23T01:07:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.127+0000 7f82f4c5f700 1 --2- 172.21.15.46:0/856378301 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82f0101b70 0x7f82f0101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.127+0000 7f82f4c5f700 1 --2- 172.21.15.46:0/856378301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f00fce60 0x7f82f0108930 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.127+0000 7f82f4c5f700 1 --2- 172.21.15.46:0/856378301 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f82f00fc4b0 0x7f82f00fc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.127+0000 7f82f4c5f700 1 -- 172.21.15.46:0/856378301 >> 172.21.15.46:0/856378301 conn(0x7f82f00713e0 msgr2=0x7f82f00717e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.127+0000 7f82f4c5f700 1 -- 172.21.15.46:0/856378301 shutdown_connections 2023-06-23T01:07:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.127+0000 7f82f4c5f700 1 -- 172.21.15.46:0/856378301 wait complete. 2023-06-23T01:07:16.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82f4c5f700 1 Processor -- start 2023-06-23T01:07:16.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82f4c5f700 1 -- start start 2023-06-23T01:07:16.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82f4c5f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f82f00fc4b0 0x7f82f010e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:16.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82f4c5f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f00fce60 0x7f82f010ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:16.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82f4c5f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82f0101b70 0x7f82f0119730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:16.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82f4c5f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f010f470 con 0x7f82f00fce60 2023-06-23T01:07:16.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82f4c5f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0119c70 con 0x7f82f0101b70 2023-06-23T01:07:16.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82f4c5f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0119db0 con 0x7f82f00fc4b0 2023-06-23T01:07:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82edd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f00fce60 0x7f82f010ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82edd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f00fce60 0x7f82f010ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47192/0 (socket says 172.21.15.46:47192) 2023-06-23T01:07:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82edd9b700 1 -- 172.21.15.46:0/1533003354 learned_addr learned my addr 172.21.15.46:0/1533003354 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82eed9d700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82f0101b70 0x7f82f0119730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82edd9b700 1 -- 172.21.15.46:0/1533003354 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f82f00fc4b0 msgr2=0x7f82f010e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82edd9b700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f82f00fc4b0 0x7f82f010e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82edd9b700 1 -- 172.21.15.46:0/1533003354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82f0101b70 msgr2=0x7f82f0119730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82edd9b700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82f0101b70 0x7f82f0119730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82edd9b700 1 -- 172.21.15.46:0/1533003354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e8009940 con 0x7f82f00fce60 2023-06-23T01:07:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.128+0000 7f82edd9b700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f00fce60 0x7f82f010ebf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f82e4002b00 tx=0x7f82e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.129+0000 7f82deffd700 1 -- 172.21.15.46:0/1533003354 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4007cc0 con 0x7f82f00fce60 2023-06-23T01:07:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.129+0000 7f82deffd700 1 -- 172.21.15.46:0/1533003354 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f82e400f040 con 0x7f82f00fce60 2023-06-23T01:07:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.129+0000 7f82f4c5f700 1 -- 172.21.15.46:0/1533003354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f0119fe0 con 0x7f82f00fce60 2023-06-23T01:07:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.129+0000 7f82deffd700 1 -- 172.21.15.46:0/1533003354 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4009430 con 0x7f82f00fce60 2023-06-23T01:07:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.129+0000 7f82f4c5f700 1 -- 172.21.15.46:0/1533003354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f011a4f0 con 0x7f82f00fce60 2023-06-23T01:07:16.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.130+0000 7f82f4c5f700 1 -- 172.21.15.46:0/1533003354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f0063d10 con 0x7f82f00fce60 2023-06-23T01:07:16.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.131+0000 7f82deffd700 1 -- 172.21.15.46:0/1533003354 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f82e400d630 con 0x7f82f00fce60 2023-06-23T01:07:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.132+0000 7f82deffd700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f82d809b700 0x7f82d809dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.132+0000 7f82ee59c700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f82d809b700 0x7f82d809dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.132+0000 7f82deffd700 1 -- 172.21.15.46:0/1533003354 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f82e40c4db0 con 0x7f82f00fce60 2023-06-23T01:07:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.132+0000 7f82ee59c700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f82d809b700 0x7f82d809dbb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f82f0065880 tx=0x7f82e0009ce0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.135+0000 7f82deffd700 1 -- 172.21.15.46:0/1533003354 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f82e4090ab0 con 0x7f82f00fce60 2023-06-23T01:07:16.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.328+0000 7f82f4c5f700 1 -- 172.21.15.46:0/1533003354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f82f011a770 con 0x7f82f00fce60 2023-06-23T01:07:16.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.328+0000 7f82deffd700 1 -- 172.21.15.46:0/1533003354 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f82e40940d0 con 0x7f82f00fce60 2023-06-23T01:07:16.332 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-06-23T01:07:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.330+0000 7f82f4c5f700 1 -- 172.21.15.46:0/1533003354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f82d809b700 msgr2=0x7f82d809dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.330+0000 7f82f4c5f700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f82d809b700 0x7f82d809dbb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f82f0065880 tx=0x7f82e0009ce0 comp rx=0 tx=0).stop 2023-06-23T01:07:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.330+0000 7f82f4c5f700 1 -- 172.21.15.46:0/1533003354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f00fce60 msgr2=0x7f82f010ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.330+0000 7f82f4c5f700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f00fce60 0x7f82f010ebf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f82e4002b00 tx=0x7f82e40046b0 comp rx=0 tx=0).stop 2023-06-23T01:07:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.330+0000 7f82f4c5f700 1 -- 172.21.15.46:0/1533003354 shutdown_connections 2023-06-23T01:07:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.330+0000 7f82f4c5f700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f82d809b700 0x7f82d809dbb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.330+0000 7f82f4c5f700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f82f0101b70 0x7f82f0119730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.330+0000 7f82f4c5f700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f82f00fc4b0 0x7f82f010e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.330+0000 7f82f4c5f700 1 --2- 172.21.15.46:0/1533003354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82f00fce60 0x7f82f010ebf0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.330+0000 7f82f4c5f700 1 -- 172.21.15.46:0/1533003354 >> 172.21.15.46:0/1533003354 conn(0x7f82f00713e0 msgr2=0x7f82f01028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.331+0000 7f82f4c5f700 1 -- 172.21.15.46:0/1533003354 shutdown_connections 2023-06-23T01:07:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:16.331+0000 7f82f4c5f700 1 -- 172.21.15.46:0/1533003354 wait complete. 2023-06-23T01:07:16.775 INFO:tasks.cephadm:Adding mds.a on smithi046 2023-06-23T01:07:16.775 INFO:tasks.cephadm:Adding mds.d on smithi046 2023-06-23T01:07:16.776 INFO:tasks.cephadm:Adding mds.g on smithi046 2023-06-23T01:07:16.776 INFO:tasks.cephadm:Adding mds.j on smithi046 2023-06-23T01:07:16.776 INFO:tasks.cephadm:Adding mds.b on smithi092 2023-06-23T01:07:16.776 INFO:tasks.cephadm:Adding mds.e on smithi092 2023-06-23T01:07:16.776 INFO:tasks.cephadm:Adding mds.h on smithi092 2023-06-23T01:07:16.777 INFO:tasks.cephadm:Adding mds.k on smithi092 2023-06-23T01:07:16.777 INFO:tasks.cephadm:Adding mds.c on smithi131 2023-06-23T01:07:16.777 INFO:tasks.cephadm:Adding mds.f on smithi131 2023-06-23T01:07:16.777 INFO:tasks.cephadm:Adding mds.i on smithi131 2023-06-23T01:07:16.777 INFO:tasks.cephadm:Adding mds.l on smithi131 2023-06-23T01:07:16.778 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:42ddc46e6e6582d8e29a5a982a81886fe028be2c shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 458642ee-1161-11ee-9b2b-001a4aab830c -- ceph orch apply mds all '12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l' 2023-06-23T01:07:17.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:16 smithi046 ceph-mon[84949]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:17.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:16 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/1533003354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-23T01:07:17.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:16 smithi131 ceph-mon[86495]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:17.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:16 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/1533003354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-23T01:07:17.378 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/mon.c/config 2023-06-23T01:07:17.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:16 smithi092 ceph-mon[85670]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:17.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:16 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/1533003354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-23T01:07:18.595 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.592+0000 7f95c1dc1700 1 -- 172.21.15.131:0/2163486642 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95bc0fbb40 msgr2=0x7f95bc0fbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:18.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.592+0000 7f95c1dc1700 1 --2- 172.21.15.131:0/2163486642 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95bc0fbb40 0x7f95bc0fbf50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0049c0 tx=0x7f95ac009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:18.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.592+0000 7f95c1dc1700 1 -- 172.21.15.131:0/2163486642 shutdown_connections 2023-06-23T01:07:18.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.592+0000 7f95c1dc1700 1 --2- 172.21.15.131:0/2163486642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95bc0fd6e0 0x7f95bc103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:18.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.592+0000 7f95c1dc1700 1 --2- 172.21.15.131:0/2163486642 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95bc0fbb40 0x7f95bc0fbf50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:18.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.592+0000 7f95c1dc1700 1 --2- 172.21.15.131:0/2163486642 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95bc0fcd50 0x7f95bc0fd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:18.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.592+0000 7f95c1dc1700 1 -- 172.21.15.131:0/2163486642 >> 172.21.15.131:0/2163486642 conn(0x7f95bc0f72a0 msgr2=0x7f95bc0f96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:18.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.592+0000 7f95c1dc1700 1 -- 172.21.15.131:0/2163486642 shutdown_connections 2023-06-23T01:07:18.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.592+0000 7f95c1dc1700 1 -- 172.21.15.131:0/2163486642 wait complete. 2023-06-23T01:07:18.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95c1dc1700 1 Processor -- start 2023-06-23T01:07:18.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95c1dc1700 1 -- start start 2023-06-23T01:07:18.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95c1dc1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95bc0fbb40 0x7f95bc10c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:18.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95c1dc1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95bc0fcd50 0x7f95bc10cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:18.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95c1dc1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95bc0fd6e0 0x7f95bc113c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:18.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95c1dc1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc114250 con 0x7f95bc0fcd50 2023-06-23T01:07:18.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95c1dc1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc1143c0 con 0x7f95bc0fbb40 2023-06-23T01:07:18.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95c1dc1700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc114580 con 0x7f95bc0fd6e0 2023-06-23T01:07:18.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95bbfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95bc0fd6e0 0x7f95bc113c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:18.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95bbfff700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95bc0fd6e0 0x7f95bc113c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.131:3300/0 says I am v2:172.21.15.131:48610/0 (socket says 172.21.15.131:48610) 2023-06-23T01:07:18.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95bbfff700 1 -- 172.21.15.131:0/1367962727 learned_addr learned my addr 172.21.15.131:0/1367962727 (peer_addr_for_me v2:172.21.15.131:0/0) 2023-06-23T01:07:18.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95bb7fe700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95bc0fbb40 0x7f95bc10c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:18.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95b3fff700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95bc0fcd50 0x7f95bc10cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:18.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95bbfff700 1 -- 172.21.15.131:0/1367962727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95bc0fbb40 msgr2=0x7f95bc10c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:18.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95bbfff700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95bc0fbb40 0x7f95bc10c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:18.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95bbfff700 1 -- 172.21.15.131:0/1367962727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95bc0fcd50 msgr2=0x7f95bc10cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:18.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95bbfff700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95bc0fcd50 0x7f95bc10cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:18.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95bbfff700 1 -- 172.21.15.131:0/1367962727 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac005040 con 0x7f95bc0fd6e0 2023-06-23T01:07:18.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.593+0000 7f95bb7fe700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95bc0fbb40 0x7f95bc10c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:07:18.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.594+0000 7f95bbfff700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95bc0fd6e0 0x7f95bc113c40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95b400cb90 tx=0x7f95b4007f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:18.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.594+0000 7f95b8ff9700 1 -- 172.21.15.131:0/1367962727 <== mon.1 v2:172.21.15.131:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b4004020 con 0x7f95bc0fd6e0 2023-06-23T01:07:18.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.594+0000 7f95b8ff9700 1 -- 172.21.15.131:0/1367962727 <== mon.1 v2:172.21.15.131:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f95b4004180 con 0x7f95bc0fd6e0 2023-06-23T01:07:18.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.594+0000 7f95b8ff9700 1 -- 172.21.15.131:0/1367962727 <== mon.1 v2:172.21.15.131:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b4003990 con 0x7f95bc0fd6e0 2023-06-23T01:07:18.601 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.594+0000 7f95c1dc1700 1 -- 172.21.15.131:0/1367962727 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc114860 con 0x7f95bc0fd6e0 2023-06-23T01:07:18.602 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.594+0000 7f95c1dc1700 1 -- 172.21.15.131:0/1367962727 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc114dd0 con 0x7f95bc0fd6e0 2023-06-23T01:07:18.602 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.595+0000 7f95c1dc1700 1 -- 172.21.15.131:0/1367962727 --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95bc107190 con 0x7f95bc0fd6e0 2023-06-23T01:07:18.602 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.596+0000 7f95b8ff9700 1 -- 172.21.15.131:0/1367962727 <== mon.1 v2:172.21.15.131:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f95b400a7a0 con 0x7f95bc0fd6e0 2023-06-23T01:07:18.603 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.596+0000 7f95b8ff9700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f95a809b700 0x7f95a809dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:18.603 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.597+0000 7f95b8ff9700 1 -- 172.21.15.131:0/1367962727 <== mon.1 v2:172.21.15.131:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f95b40c4a80 con 0x7f95bc0fd6e0 2023-06-23T01:07:18.603 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.597+0000 7f95bb7fe700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f95a809b700 0x7f95a809dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:18.603 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.597+0000 7f95bb7fe700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f95a809b700 0x7f95a809dbb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f95ac005010 tx=0x7f95ac009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:18.603 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.601+0000 7f95b8ff9700 1 -- 172.21.15.131:0/1367962727 <== mon.1 v2:172.21.15.131:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f95b40906e0 con 0x7f95bc0fd6e0 2023-06-23T01:07:18.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:18 smithi131 ceph-mon[86495]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:18.796 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.793+0000 7f95c1dc1700 1 -- 172.21.15.131:0/1367962727 --> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f95bc05df70 con 0x7f95a809b700 2023-06-23T01:07:18.804 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.801+0000 7f95b8ff9700 1 -- 172.21.15.131:0/1367962727 <== mgr.14152 v2:172.21.15.46:6800/1578446983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f95bc05df70 con 0x7f95a809b700 2023-06-23T01:07:18.804 INFO:teuthology.orchestra.run.smithi131.stdout:Scheduled mds.all update... 2023-06-23T01:07:18.806 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 -- 172.21.15.131:0/1367962727 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f95a809b700 msgr2=0x7f95a809dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:18.806 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f95a809b700 0x7f95a809dbb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f95ac005010 tx=0x7f95ac009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:18.806 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 -- 172.21.15.131:0/1367962727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95bc0fd6e0 msgr2=0x7f95bc113c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:18.807 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95bc0fd6e0 0x7f95bc113c40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95b400cb90 tx=0x7f95b4007f50 comp rx=0 tx=0).stop 2023-06-23T01:07:18.807 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 -- 172.21.15.131:0/1367962727 shutdown_connections 2023-06-23T01:07:18.807 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f95a809b700 0x7f95a809dbb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:18.808 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f95bc0fbb40 0x7f95bc10c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:18.808 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f95bc0fd6e0 0x7f95bc113c40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:18.808 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 --2- 172.21.15.131:0/1367962727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95bc0fcd50 0x7f95bc10cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:18.808 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 -- 172.21.15.131:0/1367962727 >> 172.21.15.131:0/1367962727 conn(0x7f95bc0f72a0 msgr2=0x7f95bc0f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:18.808 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 -- 172.21.15.131:0/1367962727 shutdown_connections 2023-06-23T01:07:18.808 INFO:teuthology.orchestra.run.smithi131.stderr:2023-06-23T01:07:18.803+0000 7f95c1dc1700 1 -- 172.21.15.131:0/1367962727 wait complete. 2023-06-23T01:07:18.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:18 smithi046 ceph-mon[84949]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:18.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:18 smithi092 ceph-mon[85670]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:19.224 DEBUG:teuthology.orchestra.run.smithi046:mds.a> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.a.service 2023-06-23T01:07:19.227 DEBUG:teuthology.orchestra.run.smithi046:mds.d> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.d.service 2023-06-23T01:07:19.263 INFO:journalctl@ceph.mds.a.smithi046.stdout:-- Logs begin at Fri 2023-06-23 00:44:15 UTC. -- 2023-06-23T01:07:19.264 DEBUG:teuthology.orchestra.run.smithi046:mds.g> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.g.service 2023-06-23T01:07:19.292 INFO:journalctl@ceph.mds.d.smithi046.stdout:-- Logs begin at Fri 2023-06-23 00:44:15 UTC. -- 2023-06-23T01:07:19.293 DEBUG:teuthology.orchestra.run.smithi046:mds.j> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.j.service 2023-06-23T01:07:19.319 INFO:journalctl@ceph.mds.g.smithi046.stdout:-- Logs begin at Fri 2023-06-23 00:44:15 UTC. -- 2023-06-23T01:07:19.320 DEBUG:teuthology.orchestra.run.smithi092:mds.b> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.b.service 2023-06-23T01:07:19.323 DEBUG:teuthology.orchestra.run.smithi092:mds.e> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.e.service 2023-06-23T01:07:19.346 INFO:journalctl@ceph.mds.j.smithi046.stdout:-- Logs begin at Fri 2023-06-23 00:44:15 UTC. -- 2023-06-23T01:07:19.358 INFO:journalctl@ceph.mds.b.smithi092.stdout:-- Logs begin at Fri 2023-06-23 00:44:08 UTC. -- 2023-06-23T01:07:19.359 DEBUG:teuthology.orchestra.run.smithi092:mds.h> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.h.service 2023-06-23T01:07:19.385 INFO:journalctl@ceph.mds.e.smithi092.stdout:-- Logs begin at Fri 2023-06-23 00:44:08 UTC. -- 2023-06-23T01:07:19.386 DEBUG:teuthology.orchestra.run.smithi092:mds.k> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.k.service 2023-06-23T01:07:19.410 INFO:journalctl@ceph.mds.h.smithi092.stdout:-- Logs begin at Fri 2023-06-23 00:44:08 UTC. -- 2023-06-23T01:07:19.411 DEBUG:teuthology.orchestra.run.smithi131:mds.c> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.c.service 2023-06-23T01:07:19.413 DEBUG:teuthology.orchestra.run.smithi131:mds.f> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.f.service 2023-06-23T01:07:19.435 INFO:journalctl@ceph.mds.k.smithi092.stdout:-- Logs begin at Fri 2023-06-23 00:44:08 UTC. -- 2023-06-23T01:07:19.440 INFO:journalctl@ceph.mds.c.smithi131.stdout:-- Logs begin at Fri 2023-06-23 00:44:02 UTC. -- 2023-06-23T01:07:19.441 DEBUG:teuthology.orchestra.run.smithi131:mds.i> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.i.service 2023-06-23T01:07:19.469 INFO:journalctl@ceph.mds.f.smithi131.stdout:-- Logs begin at Fri 2023-06-23 00:44:02 UTC. -- 2023-06-23T01:07:19.471 DEBUG:teuthology.orchestra.run.smithi131:mds.l> sudo journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.l.service 2023-06-23T01:07:19.494 INFO:journalctl@ceph.mds.i.smithi131.stdout:-- Logs begin at Fri 2023-06-23 00:44:02 UTC. -- 2023-06-23T01:07:19.495 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-06-23T01:07:19.496 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:07:19.520 INFO:journalctl@ceph.mds.l.smithi131.stdout:-- Logs begin at Fri 2023-06-23 00:44:02 UTC. -- 2023-06-23T01:07:19.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.651+0000 7f1c61736700 1 -- 172.21.15.46:0/2530814349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5c0fccb0 msgr2=0x7f1c5c0fd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.651+0000 7f1c61736700 1 --2- 172.21.15.46:0/2530814349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5c0fccb0 0x7f1c5c0fd0c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1c500049c0 tx=0x7f1c50009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.651+0000 7f1c61736700 1 -- 172.21.15.46:0/2530814349 shutdown_connections 2023-06-23T01:07:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.651+0000 7f1c61736700 1 --2- 172.21.15.46:0/2530814349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5c0fe720 0x7f1c5c105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.651+0000 7f1c61736700 1 --2- 172.21.15.46:0/2530814349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5c0fccb0 0x7f1c5c0fd0c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.651+0000 7f1c61736700 1 --2- 172.21.15.46:0/2530814349 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c5c0fd600 0x7f1c5c0fdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.651+0000 7f1c61736700 1 -- 172.21.15.46:0/2530814349 >> 172.21.15.46:0/2530814349 conn(0x7f1c5c0f84a0 msgr2=0x7f1c5c0fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.651+0000 7f1c61736700 1 -- 172.21.15.46:0/2530814349 shutdown_connections 2023-06-23T01:07:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.651+0000 7f1c61736700 1 -- 172.21.15.46:0/2530814349 wait complete. 2023-06-23T01:07:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.651+0000 7f1c61736700 1 Processor -- start 2023-06-23T01:07:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.651+0000 7f1c61736700 1 -- start start 2023-06-23T01:07:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c61736700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5c0fccb0 0x7f1c5c10d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c61736700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5c0fd600 0x7f1c5c10ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c61736700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c5c0fe720 0x7f1c5c114e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c61736700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5c115470 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c61736700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5c1155e0 con 0x7f1c5c0fd600 2023-06-23T01:07:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c61736700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c5c1157a0 con 0x7f1c5c0fe720 2023-06-23T01:07:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c5affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5c0fccb0 0x7f1c5c10d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c5affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5c0fccb0 0x7f1c5c10d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47218/0 (socket says 172.21.15.46:47218) 2023-06-23T01:07:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c5affd700 1 -- 172.21.15.46:0/1174922004 learned_addr learned my addr 172.21.15.46:0/1174922004 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c5a7fc700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5c0fd600 0x7f1c5c10ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c5b7fe700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c5c0fe720 0x7f1c5c114e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c5affd700 1 -- 172.21.15.46:0/1174922004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c5c0fe720 msgr2=0x7f1c5c114e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:19.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c5affd700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c5c0fe720 0x7f1c5c114e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c5affd700 1 -- 172.21.15.46:0/1174922004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5c0fd600 msgr2=0x7f1c5c10ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c5affd700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5c0fd600 0x7f1c5c10ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c5affd700 1 -- 172.21.15.46:0/1174922004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c50005040 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c5affd700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5c0fccb0 0x7f1c5c10d890 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1c5000b920 tx=0x7f1c50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c437fe700 1 -- 172.21.15.46:0/1174922004 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5000e040 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c437fe700 1 -- 172.21.15.46:0/1174922004 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1c50009570 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c61736700 1 -- 172.21.15.46:0/1174922004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5c115a20 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c437fe700 1 -- 172.21.15.46:0/1174922004 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c50013cc0 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.652+0000 7f1c61736700 1 -- 172.21.15.46:0/1174922004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5c115f00 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.653+0000 7f1c61736700 1 -- 172.21.15.46:0/1174922004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c5c103cc0 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.654+0000 7f1c437fe700 1 -- 172.21.15.46:0/1174922004 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f1c500134b0 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.655+0000 7f1c437fe700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1c4409b680 0x7f1c4409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.655+0000 7f1c437fe700 1 -- 172.21.15.46:0/1174922004 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f1c500c5a70 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.655+0000 7f1c5a7fc700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1c4409b680 0x7f1c4409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.655+0000 7f1c5a7fc700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1c4409b680 0x7f1c4409db30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009b80 tx=0x7f1c4c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:19.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.659+0000 7f1c437fe700 1 -- 172.21.15.46:0/1174922004 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f1c50091770 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.856+0000 7f1c61736700 1 -- 172.21.15.46:0/1174922004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c5c108570 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.857+0000 7f1c437fe700 1 -- 172.21.15.46:0/1174922004 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18097 (secure 0 0 0) 0x7f1c50094d90 con 0x7f1c5c0fccb0 2023-06-23T01:07:19.860 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:19.860 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":77,"fsid":"458642ee-1161-11ee-9b2b-001a4aab830c","created":"2023-06-23T01:00:57.877538+0000","modified":"2023-06-23T01:07:09.444468+0000","last_up_change":"2023-06-23T01:07:07.914869+0000","last_in_change":"2023-06-23T01:06:51.140023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:04:00.299796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f28767ce-a780-4e50-b118-37a8c275c0e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6803","nonce":709285010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6805","nonce":709285010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6809","nonce":709285010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6807","nonce":709285010}]},"public_addr":"172.21.15.46:6803/709285010","cluster_addr":"172.21.15.46:6805/709285010","heartbeat_back_addr":"172.21.15.46:6809/709285010","heartbeat_front_addr":"172.21.15.46:6807/709285010","state":["exists","up"]},{"osd":1,"uuid":"4d4c4c36-d4a1-4b96-9059-adffee04401a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6801","nonce":4266072183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6803","nonce":4266072183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6807","nonce":4266072183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6805","nonce":4266072183}]},"public_addr":"172.21.15.92:6801/4266072183","cluster_addr":"172.21.15.92:6803/4266072183","heartbeat_back_addr":"172.21.15.92:6807/4266072183","heartbeat_front_addr":"172.21.15.92:6805/4266072183","state":["exists","up"]},{"osd":2,"uuid":"ec7b0c60-88ea-4c2b-a34c-96d188e02014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6801","nonce":1787957441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6803","nonce":1787957441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6807","nonce":1787957441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6805","nonce":1787957441}]},"public_addr":"172.21.15.131:6801/1787957441","cluster_addr":"172.21.15.131:6803/1787957441","heartbeat_back_addr":"172.21.15.131:6807/1787957441","heartbeat_front_addr":"172.21.15.131:6805/1787957441","state":["exists","up"]},{"osd":3,"uuid":"32670a23-3286-4538-9628-38d763429cbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6811","nonce":338690444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6813","nonce":338690444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6817","nonce":338690444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6815","nonce":338690444}]},"public_addr":"172.21.15.46:6811/338690444","cluster_addr":"172.21.15.46:6813/338690444","heartbeat_back_addr":"172.21.15.46:6817/338690444","heartbeat_front_addr":"172.21.15.46:6815/338690444","state":["exists","up"]},{"osd":4,"uuid":"0d8f982c-eb74-48a2-949e-d0ccd7732937","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6809","nonce":2024629902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6811","nonce":2024629902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6815","nonce":2024629902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6813","nonce":2024629902}]},"public_addr":"172.21.15.92:6809/2024629902","cluster_addr":"172.21.15.92:6811/2024629902","heartbeat_back_addr":"172.21.15.92:6815/2024629902","heartbeat_front_addr":"172.21.15.92:6813/2024629902","state":["exists","up"]},{"osd":5,"uuid":"52a7160c-4735-4a77-bc08-4d411c282b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6809","nonce":2577961971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6811","nonce":2577961971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6815","nonce":2577961971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6813","nonce":2577961971}]},"public_addr":"172.21.15.131:6809/2577961971","cluster_addr":"172.21.15.131:6811/2577961971","heartbeat_back_addr":"172.21.15.131:6815/2577961971","heartbeat_front_addr":"172.21.15.131:6813/2577961971","state":["exists","up"]},{"osd":6,"uuid":"3f3d2690-f655-4dda-a36c-bc97f9d08a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6819","nonce":558288790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6821","nonce":558288790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6825","nonce":558288790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6823","nonce":558288790}]},"public_addr":"172.21.15.46:6819/558288790","cluster_addr":"172.21.15.46:6821/558288790","heartbeat_back_addr":"172.21.15.46:6825/558288790","heartbeat_front_addr":"172.21.15.46:6823/558288790","state":["exists","up"]},{"osd":7,"uuid":"2eb43609-8921-4891-b0c4-af932bfb69bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6817","nonce":3796540277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6819","nonce":3796540277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6823","nonce":3796540277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6821","nonce":3796540277}]},"public_addr":"172.21.15.92:6817/3796540277","cluster_addr":"172.21.15.92:6819/3796540277","heartbeat_back_addr":"172.21.15.92:6823/3796540277","heartbeat_front_addr":"172.21.15.92:6821/3796540277","state":["exists","up"]},{"osd":8,"uuid":"0139b39e-f933-4685-b056-a6c27fa87fd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6817","nonce":1004740496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6819","nonce":1004740496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6823","nonce":1004740496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6821","nonce":1004740496}]},"public_addr":"172.21.15.131:6817/1004740496","cluster_addr":"172.21.15.131:6819/1004740496","heartbeat_back_addr":"172.21.15.131:6823/1004740496","heartbeat_front_addr":"172.21.15.131:6821/1004740496","state":["exists","up"]},{"osd":9,"uuid":"8bd083b4-f766-4e1b-b959-19d2fa82d77f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6827","nonce":3042170676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6829","nonce":3042170676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6833","nonce":3042170676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6831","nonce":3042170676}]},"public_addr":"172.21.15.46:6827/3042170676","cluster_addr":"172.21.15.46:6829/3042170676","heartbeat_back_addr":"172.21.15.46:6833/3042170676","heartbeat_front_addr":"172.21.15.46:6831/3042170676","state":["exists","up"]},{"osd":10,"uuid":"0b4f58f5-d971-4634-a1b6-896501d3d218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6825","nonce":2375910823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6827","nonce":2375910823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6831","nonce":2375910823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6829","nonce":2375910823}]},"public_addr":"172.21.15.92:6825/2375910823","cluster_addr":"172.21.15.92:6827/2375910823","heartbeat_back_addr":"172.21.15.92:6831/2375910823","heartbeat_front_addr":"172.21.15.92:6829/2375910823","state":["exists","up"]},{"osd":11,"uuid":"7d7a4115-c6e9-4cef-865b-05f57fbcface","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6825","nonce":1716414800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6827","nonce":1716414800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6831","nonce":1716414800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6829","nonce":1716414800}]},"public_addr":"172.21.15.131:6825/1716414800","cluster_addr":"172.21.15.131:6827/1716414800","heartbeat_back_addr":"172.21.15.131:6831/1716414800","heartbeat_front_addr":"172.21.15.131:6829/1716414800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:15.785940+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:36.479807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:57.678238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:04:20.396101+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:04:41.179881+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:01.570524+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:22.313074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:43.129445+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:03.897470+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:25.196569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:46.208577+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:07:06.788656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/685567953":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6801/2331157358":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6800/2331157358":"2023-06-24T01:01:55.786463+0000","172.21.15.46:0/1655911006":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/187730254":"2023-06-24T01:01:32.095580+0000","172.21.15.46:6801/3291477171":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/3127678959":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6800/3291477171":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/411746526":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/360533287":"2023-06-24T01:01:55.786463+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:07:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 -- 172.21.15.46:0/1174922004 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1c4409b680 msgr2=0x7f1c4409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1c4409b680 0x7f1c4409db30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009b80 tx=0x7f1c4c00b040 comp rx=0 tx=0).stop 2023-06-23T01:07:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 -- 172.21.15.46:0/1174922004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5c0fccb0 msgr2=0x7f1c5c10d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5c0fccb0 0x7f1c5c10d890 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1c5000b920 tx=0x7f1c50009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 -- 172.21.15.46:0/1174922004 shutdown_connections 2023-06-23T01:07:19.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1c5c0fe720 0x7f1c5c114e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5c0fccb0 0x7f1c5c10d890 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1c4409b680 0x7f1c4409db30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 --2- 172.21.15.46:0/1174922004 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1c5c0fd600 0x7f1c5c10ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 -- 172.21.15.46:0/1174922004 >> 172.21.15.46:0/1174922004 conn(0x7f1c5c0f84a0 msgr2=0x7f1c5c0fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 -- 172.21.15.46:0/1174922004 shutdown_connections 2023-06-23T01:07:19.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.859+0000 7f1c61736700 1 -- 172.21.15.46:0/1174922004 wait complete. 2023-06-23T01:07:19.876 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-23T01:04:00.299796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-23T01:07:19.877 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T01:07:19.972 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:19 smithi131 ceph-mon[86495]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:07:19.972 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:19 smithi131 ceph-mon[86495]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l;count:12 2023-06-23T01:07:19.972 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:19 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:19.973 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:19 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:19.973 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:19 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:19.973 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:19 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:19.974 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:19 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:19.974 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:19 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:19.974 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:19 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:19.974 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:19 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:19.975 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:19 smithi131 ceph-mon[86495]: Deploying daemon mds.l on smithi131 2023-06-23T01:07:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.997+0000 7f382f59e700 1 -- 172.21.15.46:0/2862689008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38300fb9c0 msgr2=0x7f38300fbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.997+0000 7f382f59e700 1 --2- 172.21.15.46:0/2862689008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38300fb9c0 0x7f38300fbdd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f38200049c0 tx=0x7f3820009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.997+0000 7f382f59e700 1 -- 172.21.15.46:0/2862689008 shutdown_connections 2023-06-23T01:07:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.997+0000 7f382f59e700 1 --2- 172.21.15.46:0/2862689008 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38300fd620 0x7f3830103ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.997+0000 7f382f59e700 1 --2- 172.21.15.46:0/2862689008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38300fb9c0 0x7f38300fbdd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.997+0000 7f382f59e700 1 --2- 172.21.15.46:0/2862689008 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38300fcc70 0x7f38300fd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.997+0000 7f382f59e700 1 -- 172.21.15.46:0/2862689008 >> 172.21.15.46:0/2862689008 conn(0x7f38300f7100 msgr2=0x7f38300f9550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.997+0000 7f382f59e700 1 -- 172.21.15.46:0/2862689008 shutdown_connections 2023-06-23T01:07:20.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.997+0000 7f382f59e700 1 -- 172.21.15.46:0/2862689008 wait complete. 2023-06-23T01:07:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382f59e700 1 Processor -- start 2023-06-23T01:07:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382f59e700 1 -- start start 2023-06-23T01:07:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382f59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38300fb9c0 0x7f3830065a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382f59e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38300fcc70 0x7f3830112480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382f59e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38300fd620 0x7f3830114ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382f59e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3830115260 con 0x7f38300fcc70 2023-06-23T01:07:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382f59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38301153d0 con 0x7f38300fb9c0 2023-06-23T01:07:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382f59e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3830115590 con 0x7f38300fd620 2023-06-23T01:07:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38300fcc70 0x7f3830112480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38300fcc70 0x7f3830112480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47246/0 (socket says 172.21.15.46:47246) 2023-06-23T01:07:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382dd9b700 1 -- 172.21.15.46:0/4176389263 learned_addr learned my addr 172.21.15.46:0/4176389263 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382ed9d700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38300fd620 0x7f3830114ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382e59c700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38300fb9c0 0x7f3830065a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382dd9b700 1 -- 172.21.15.46:0/4176389263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38300fd620 msgr2=0x7f3830114ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382dd9b700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38300fd620 0x7f3830114ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382dd9b700 1 -- 172.21.15.46:0/4176389263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38300fb9c0 msgr2=0x7f3830065a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382dd9b700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38300fb9c0 0x7f3830065a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382dd9b700 1 -- 172.21.15.46:0/4176389263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3820005040 con 0x7f38300fcc70 2023-06-23T01:07:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.998+0000 7f382dd9b700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38300fcc70 0x7f3830112480 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3824002ad0 tx=0x7f38240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.999+0000 7f381effd700 1 -- 172.21.15.46:0/4176389263 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824007cc0 con 0x7f38300fcc70 2023-06-23T01:07:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.999+0000 7f381effd700 1 -- 172.21.15.46:0/4176389263 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f382400f040 con 0x7f38300fcc70 2023-06-23T01:07:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.999+0000 7f382f59e700 1 -- 172.21.15.46:0/4176389263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3830115870 con 0x7f38300fcc70 2023-06-23T01:07:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.999+0000 7f381effd700 1 -- 172.21.15.46:0/4176389263 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382400d880 con 0x7f38300fcc70 2023-06-23T01:07:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:19.999+0000 7f382f59e700 1 -- 172.21.15.46:0/4176389263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3830115d60 con 0x7f38300fcc70 2023-06-23T01:07:20.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.000+0000 7f382f59e700 1 -- 172.21.15.46:0/4176389263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3830107080 con 0x7f38300fcc70 2023-06-23T01:07:20.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.001+0000 7f381effd700 1 -- 172.21.15.46:0/4176389263 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f382400dcb0 con 0x7f38300fcc70 2023-06-23T01:07:20.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.001+0000 7f381effd700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f381809b680 0x7f381809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:20.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.001+0000 7f382e59c700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f381809b680 0x7f381809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:20.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.001+0000 7f381effd700 1 -- 172.21.15.46:0/4176389263 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f38240c4ce0 con 0x7f38300fcc70 2023-06-23T01:07:20.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.002+0000 7f382e59c700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f381809b680 0x7f381809db30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3820009b80 tx=0x7f3820009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:20.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.004+0000 7f381effd700 1 -- 172.21.15.46:0/4176389263 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f38240909e0 con 0x7f38300fcc70 2023-06-23T01:07:20.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:19 smithi046 ceph-mon[84949]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:07:20.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:19 smithi046 ceph-mon[84949]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l;count:12 2023-06-23T01:07:20.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:20.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:20.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:20.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:20.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:20.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:20.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:20.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:19 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:20.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:19 smithi046 ceph-mon[84949]: Deploying daemon mds.l on smithi131 2023-06-23T01:07:20.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:19 smithi092 ceph-mon[85670]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l", "target": ["mon-mgr", ""]}]: dispatch 2023-06-23T01:07:20.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:19 smithi092 ceph-mon[85670]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi092=b;smithi092=e;smithi092=h;smithi092=k;smithi131=c;smithi131=f;smithi131=i;smithi131=l;count:12 2023-06-23T01:07:20.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:19 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:20.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:19 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:20.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:19 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:20.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:19 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:20.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:19 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:20.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:19 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:20.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:19 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:20.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:19 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:20.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:19 smithi092 ceph-mon[85670]: Deploying daemon mds.l on smithi131 2023-06-23T01:07:20.202 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.200+0000 7f382f59e700 1 -- 172.21.15.46:0/4176389263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3830115f80 con 0x7f38300fcc70 2023-06-23T01:07:20.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.200+0000 7f381effd700 1 -- 172.21.15.46:0/4176389263 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3824094000 con 0x7f38300fcc70 2023-06-23T01:07:20.203 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-06-23T01:07:20.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.202+0000 7f382f59e700 1 -- 172.21.15.46:0/4176389263 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f381809b680 msgr2=0x7f381809db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:20.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.202+0000 7f382f59e700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f381809b680 0x7f381809db30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3820009b80 tx=0x7f3820009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:20.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.202+0000 7f382f59e700 1 -- 172.21.15.46:0/4176389263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38300fcc70 msgr2=0x7f3830112480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:20.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.202+0000 7f382f59e700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38300fcc70 0x7f3830112480 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3824002ad0 tx=0x7f38240046b0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.203+0000 7f382f59e700 1 -- 172.21.15.46:0/4176389263 shutdown_connections 2023-06-23T01:07:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.203+0000 7f382f59e700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f381809b680 0x7f381809db30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.203+0000 7f382f59e700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f38300fb9c0 0x7f3830065a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.203+0000 7f382f59e700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f38300fd620 0x7f3830114ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.203+0000 7f382f59e700 1 --2- 172.21.15.46:0/4176389263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38300fcc70 0x7f3830112480 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.203+0000 7f382f59e700 1 -- 172.21.15.46:0/4176389263 >> 172.21.15.46:0/4176389263 conn(0x7f38300f7100 msgr2=0x7f38300f9320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.203+0000 7f382f59e700 1 -- 172.21.15.46:0/4176389263 shutdown_connections 2023-06-23T01:07:20.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.203+0000 7f382f59e700 1 -- 172.21.15.46:0/4176389263 wait complete. 2023-06-23T01:07:20.218 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:20.232 INFO:journalctl@ceph.mds.l.smithi131.stdout:Jun 23 01:07:20 smithi131 systemd[1]: Starting Ceph mds.l for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:20.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.335+0000 7f536f638700 1 -- 172.21.15.46:0/2703387756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53680fd990 msgr2=0x7f5368104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:20.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.335+0000 7f536f638700 1 --2- 172.21.15.46:0/2703387756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53680fd990 0x7f5368104230 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5360009d60 tx=0x7f5360004d10 comp rx=0 tx=0).stop 2023-06-23T01:07:20.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.335+0000 7f536f638700 1 -- 172.21.15.46:0/2703387756 shutdown_connections 2023-06-23T01:07:20.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.335+0000 7f536f638700 1 --2- 172.21.15.46:0/2703387756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53680fd990 0x7f5368104230 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.335+0000 7f536f638700 1 --2- 172.21.15.46:0/2703387756 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f53680fbe70 0x7f53680fc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.335+0000 7f536f638700 1 --2- 172.21.15.46:0/2703387756 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53680fcfe0 0x7f53680fd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.335+0000 7f536f638700 1 -- 172.21.15.46:0/2703387756 >> 172.21.15.46:0/2703387756 conn(0x7f53680f74f0 msgr2=0x7f53680f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:20.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536f638700 1 -- 172.21.15.46:0/2703387756 shutdown_connections 2023-06-23T01:07:20.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536f638700 1 -- 172.21.15.46:0/2703387756 wait complete. 2023-06-23T01:07:20.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536f638700 1 Processor -- start 2023-06-23T01:07:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536f638700 1 -- start start 2023-06-23T01:07:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536f638700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53680fbe70 0x7f536810faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536f638700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53680fcfe0 0x7f536810ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536f638700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f53680fd990 0x7f5368117070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536f638700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5368117680 con 0x7f53680fcfe0 2023-06-23T01:07:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536f638700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53681177f0 con 0x7f53680fbe70 2023-06-23T01:07:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536f638700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53681179b0 con 0x7f53680fd990 2023-06-23T01:07:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536cbd3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53680fcfe0 0x7f536810ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:20.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536cbd3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53680fcfe0 0x7f536810ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47266/0 (socket says 172.21.15.46:47266) 2023-06-23T01:07:20.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536cbd3700 1 -- 172.21.15.46:0/2316768342 learned_addr learned my addr 172.21.15.46:0/2316768342 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:20.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536dbd5700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f53680fd990 0x7f5368117070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:20.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.336+0000 7f536d3d4700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53680fbe70 0x7f536810faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:20.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.337+0000 7f536cbd3700 1 -- 172.21.15.46:0/2316768342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f53680fd990 msgr2=0x7f5368117070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:20.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.337+0000 7f536cbd3700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f53680fd990 0x7f5368117070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.337+0000 7f536cbd3700 1 -- 172.21.15.46:0/2316768342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53680fbe70 msgr2=0x7f536810faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.337+0000 7f536cbd3700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53680fbe70 0x7f536810faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.337+0000 7f536cbd3700 1 -- 172.21.15.46:0/2316768342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5360009940 con 0x7f53680fcfe0 2023-06-23T01:07:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.337+0000 7f536cbd3700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53680fcfe0 0x7f536810ffe0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f536400ebc0 tx=0x7f536400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.337+0000 7f535dffb700 1 -- 172.21.15.46:0/2316768342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364007cc0 con 0x7f53680fcfe0 2023-06-23T01:07:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.337+0000 7f535dffb700 1 -- 172.21.15.46:0/2316768342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f536400f040 con 0x7f53680fcfe0 2023-06-23T01:07:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.337+0000 7f536f638700 1 -- 172.21.15.46:0/2316768342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5368117c30 con 0x7f53680fcfe0 2023-06-23T01:07:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.337+0000 7f535dffb700 1 -- 172.21.15.46:0/2316768342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536400c820 con 0x7f53680fcfe0 2023-06-23T01:07:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.337+0000 7f536f638700 1 -- 172.21.15.46:0/2316768342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5368118120 con 0x7f53680fcfe0 2023-06-23T01:07:20.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.338+0000 7f536f638700 1 -- 172.21.15.46:0/2316768342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5368101b40 con 0x7f53680fcfe0 2023-06-23T01:07:20.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.339+0000 7f535dffb700 1 -- 172.21.15.46:0/2316768342 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f536400cc50 con 0x7f53680fcfe0 2023-06-23T01:07:20.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.339+0000 7f535dffb700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f535409b700 0x7f535409dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:20.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.340+0000 7f535dffb700 1 -- 172.21.15.46:0/2316768342 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f53640c4500 con 0x7f53680fcfe0 2023-06-23T01:07:20.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.340+0000 7f536d3d4700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f535409b700 0x7f535409dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:20.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.340+0000 7f536d3d4700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f535409b700 0x7f535409dbb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5358009b80 tx=0x7f535800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:20.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.344+0000 7f535dffb700 1 -- 172.21.15.46:0/2316768342 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f536404da00 con 0x7f53680fcfe0 2023-06-23T01:07:20.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.587+0000 7f536f638700 1 -- 172.21.15.46:0/2316768342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53680644e0 con 0x7f53680fcfe0 2023-06-23T01:07:20.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.587+0000 7f535dffb700 1 -- 172.21.15.46:0/2316768342 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f5364090160 con 0x7f53680fcfe0 2023-06-23T01:07:20.590 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:20.590 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-06-23T01:07:20.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 -- 172.21.15.46:0/2316768342 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f535409b700 msgr2=0x7f535409dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:20.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f535409b700 0x7f535409dbb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5358009b80 tx=0x7f535800b040 comp rx=0 tx=0).stop 2023-06-23T01:07:20.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 -- 172.21.15.46:0/2316768342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53680fcfe0 msgr2=0x7f536810ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:20.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53680fcfe0 0x7f536810ffe0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f536400ebc0 tx=0x7f536400b040 comp rx=0 tx=0).stop 2023-06-23T01:07:20.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 -- 172.21.15.46:0/2316768342 shutdown_connections 2023-06-23T01:07:20.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f535409b700 0x7f535409dbb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53680fbe70 0x7f536810faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f53680fd990 0x7f5368117070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 --2- 172.21.15.46:0/2316768342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53680fcfe0 0x7f536810ffe0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:20.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 -- 172.21.15.46:0/2316768342 >> 172.21.15.46:0/2316768342 conn(0x7f53680f74f0 msgr2=0x7f53680f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:20.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 -- 172.21.15.46:0/2316768342 shutdown_connections 2023-06-23T01:07:20.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:20.590+0000 7f536f638700 1 -- 172.21.15.46:0/2316768342 wait complete. 2023-06-23T01:07:20.594 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 1 2023-06-23T01:07:21.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:20 smithi046 ceph-mon[84949]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:21.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:20 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/1174922004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:21.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:20 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/4176389263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-23T01:07:21.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:20 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2316768342' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:21.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:20 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.131:6832/3255240190,v1:172.21.15.131:6833/3255240190] up:boot 2023-06-23T01:07:21.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:20 smithi046 ceph-mon[84949]: fsmap 1 up:standby 2023-06-23T01:07:21.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:20 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-06-23T01:07:21.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:20 smithi092 ceph-mon[85670]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:21.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:20 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/1174922004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:21.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:20 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/4176389263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-23T01:07:21.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:20 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2316768342' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:21.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:20 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.131:6832/3255240190,v1:172.21.15.131:6833/3255240190] up:boot 2023-06-23T01:07:21.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:20 smithi092 ceph-mon[85670]: fsmap 1 up:standby 2023-06-23T01:07:21.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:20 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-06-23T01:07:21.232 INFO:journalctl@ceph.mds.l.smithi131.stdout:Jun 23 01:07:20 smithi131 ceph-mds[119029]: -- 172.21.15.131:0/2058967513 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d0dd4009c0 con 0x55d0dd3d6800 2023-06-23T01:07:21.232 INFO:journalctl@ceph.mds.l.smithi131.stdout:Jun 23 01:07:20 smithi131 conmon[119005]: starting mds.l at 2023-06-23T01:07:21.232 INFO:journalctl@ceph.mds.l.smithi131.stdout:Jun 23 01:07:20 smithi131 bash[118965]: da710044a1418baf0b6855badf1969e15a2a44f506e794e5fade5c655d5090b3 2023-06-23T01:07:21.232 INFO:journalctl@ceph.mds.l.smithi131.stdout:Jun 23 01:07:20 smithi131 systemd[1]: Started Ceph mds.l for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:21.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:20 smithi131 ceph-mon[86495]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:21.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:20 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/1174922004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:21.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:20 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/4176389263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-23T01:07:21.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:20 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2316768342' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:21.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:20 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.131:6832/3255240190,v1:172.21.15.131:6833/3255240190] up:boot 2023-06-23T01:07:21.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:20 smithi131 ceph-mon[86495]: fsmap 1 up:standby 2023-06-23T01:07:21.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:20 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-06-23T01:07:22.137 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:22.137 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:22.137 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:22.138 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:22.138 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:22.138 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:21 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:22.138 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:21 smithi092 ceph-mon[85670]: Deploying daemon mds.k on smithi092 2023-06-23T01:07:22.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:22.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:22.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:22.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:22.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:22.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:21 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:22.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:21 smithi131 ceph-mon[86495]: Deploying daemon mds.k on smithi092 2023-06-23T01:07:22.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:22.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:22.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:22.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:22.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:22.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:21 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:22.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:21 smithi046 ceph-mon[84949]: Deploying daemon mds.k on smithi092 2023-06-23T01:07:22.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.739+0000 7f5c91973700 1 -- 172.21.15.46:0/2831288880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c0fcc80 msgr2=0x7f5c8c0fd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.739+0000 7f5c91973700 1 --2- 172.21.15.46:0/2831288880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c0fcc80 0x7f5c8c0fd0f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5c78004900 tx=0x7f5c78009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.739+0000 7f5c91973700 1 -- 172.21.15.46:0/2831288880 shutdown_connections 2023-06-23T01:07:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.739+0000 7f5c91973700 1 --2- 172.21.15.46:0/2831288880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c8c0fd630 0x7f5c8c103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.739+0000 7f5c91973700 1 --2- 172.21.15.46:0/2831288880 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5c8c0fb9d0 0x7f5c8c0fbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.739+0000 7f5c91973700 1 --2- 172.21.15.46:0/2831288880 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c0fcc80 0x7f5c8c0fd0f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.739+0000 7f5c91973700 1 -- 172.21.15.46:0/2831288880 >> 172.21.15.46:0/2831288880 conn(0x7f5c8c0f7130 msgr2=0x7f5c8c0f9560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.739+0000 7f5c91973700 1 -- 172.21.15.46:0/2831288880 shutdown_connections 2023-06-23T01:07:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.739+0000 7f5c91973700 1 -- 172.21.15.46:0/2831288880 wait complete. 2023-06-23T01:07:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.739+0000 7f5c91973700 1 Processor -- start 2023-06-23T01:07:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.739+0000 7f5c91973700 1 -- start start 2023-06-23T01:07:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c91973700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c0fb9d0 0x7f5c8c0669a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c91973700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c8c0fcc80 0x7f5c8c1124d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c91973700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5c8c0fd630 0x7f5c8c114af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c91973700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c1152e0 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c91973700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c115450 con 0x7f5c8c0fb9d0 2023-06-23T01:07:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c91973700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c115610 con 0x7f5c8c0fd630 2023-06-23T01:07:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c8bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c8c0fcc80 0x7f5c8c1124d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c8bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c8c0fcc80 0x7f5c8c1124d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47284/0 (socket says 172.21.15.46:47284) 2023-06-23T01:07:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c8bfff700 1 -- 172.21.15.46:0/188884556 learned_addr learned my addr 172.21.15.46:0/188884556 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c90971700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c0fb9d0 0x7f5c8c0669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c91172700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5c8c0fd630 0x7f5c8c114af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c8bfff700 1 -- 172.21.15.46:0/188884556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5c8c0fd630 msgr2=0x7f5c8c114af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c8bfff700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5c8c0fd630 0x7f5c8c114af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c8bfff700 1 -- 172.21.15.46:0/188884556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c0fb9d0 msgr2=0x7f5c8c0669a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c8bfff700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c0fb9d0 0x7f5c8c0669a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c8bfff700 1 -- 172.21.15.46:0/188884556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c84009940 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c8bfff700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c8c0fcc80 0x7f5c8c1124d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5c78013e40 tx=0x7f5c78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c897fa700 1 -- 172.21.15.46:0/188884556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7800e070 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c897fa700 1 -- 172.21.15.46:0/188884556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5c78009640 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.740+0000 7f5c91973700 1 -- 172.21.15.46:0/188884556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c78005040 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.741+0000 7f5c897fa700 1 -- 172.21.15.46:0/188884556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7801ca90 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.741+0000 7f5c91973700 1 -- 172.21.15.46:0/188884556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c115bb0 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.742+0000 7f5c91973700 1 -- 172.21.15.46:0/188884556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c1070e0 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.743+0000 7f5c897fa700 1 -- 172.21.15.46:0/188884556 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f5c780091a0 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.743+0000 7f5c897fa700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5c7c09b680 0x7f5c7c09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:22.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.743+0000 7f5c90971700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5c7c09b680 0x7f5c7c09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:22.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.743+0000 7f5c897fa700 1 -- 172.21.15.46:0/188884556 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f5c780c5dc0 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.744+0000 7f5c90971700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5c7c09b680 0x7f5c7c09db30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c80004f40 tx=0x7f5c8000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.748+0000 7f5c897fa700 1 -- 172.21.15.46:0/188884556 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f5c78091ac0 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.890 INFO:journalctl@ceph.mds.k.smithi092.stdout:Jun 23 01:07:22 smithi092 systemd[1]: Starting Ceph mds.k for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:22.934 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:22 smithi046 ceph-mon[84949]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:22.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.989+0000 7f5c91973700 1 -- 172.21.15.46:0/188884556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c8c115dd0 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.989+0000 7f5c897fa700 1 -- 172.21.15.46:0/188884556 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7f5c780c5070 con 0x7f5c8c0fcc80 2023-06-23T01:07:22.992 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:22.992 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/3255240190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":3255240190},{"type":"v1","addr":"172.21.15.131:6833","nonce":3255240190}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-06-23T01:07:22.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 -- 172.21.15.46:0/188884556 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5c7c09b680 msgr2=0x7f5c7c09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:22.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5c7c09b680 0x7f5c7c09db30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c80004f40 tx=0x7f5c8000b040 comp rx=0 tx=0).stop 2023-06-23T01:07:22.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 -- 172.21.15.46:0/188884556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c8c0fcc80 msgr2=0x7f5c8c1124d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:22.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c8c0fcc80 0x7f5c8c1124d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5c78013e40 tx=0x7f5c78009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:22.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 -- 172.21.15.46:0/188884556 shutdown_connections 2023-06-23T01:07:22.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f5c7c09b680 0x7f5c7c09db30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:22.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5c8c0fb9d0 0x7f5c8c0669a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:22.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5c8c0fd630 0x7f5c8c114af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:22.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 --2- 172.21.15.46:0/188884556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c8c0fcc80 0x7f5c8c1124d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:22.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 -- 172.21.15.46:0/188884556 >> 172.21.15.46:0/188884556 conn(0x7f5c8c0f7130 msgr2=0x7f5c8c0f93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:22.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 -- 172.21.15.46:0/188884556 shutdown_connections 2023-06-23T01:07:22.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:22.991+0000 7f5c91973700 1 -- 172.21.15.46:0/188884556 wait complete. 2023-06-23T01:07:22.996 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2023-06-23T01:07:23.195 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:22 smithi092 ceph-mon[85670]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:23.196 INFO:journalctl@ceph.mds.k.smithi092.stdout:Jun 23 01:07:23 smithi092 ceph-mds[116606]: -- 172.21.15.92:0/322740170 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55afbcc7e9c0 con 0x55afbcc54800 2023-06-23T01:07:23.196 INFO:journalctl@ceph.mds.k.smithi092.stdout:Jun 23 01:07:23 smithi092 conmon[116582]: starting mds.k at 2023-06-23T01:07:23.196 INFO:journalctl@ceph.mds.k.smithi092.stdout:Jun 23 01:07:23 smithi092 bash[116542]: e5f71c6f753c2869309d6723e8e68a5dbab31fd67fb1d8f36fe4b033a9693ab0 2023-06-23T01:07:23.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:22 smithi131 ceph-mon[86495]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:23.639 INFO:journalctl@ceph.mds.k.smithi092.stdout:Jun 23 01:07:23 smithi092 systemd[1]: Started Ceph mds.k for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:24.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:23 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/188884556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:24.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:24.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:24.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:24.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:24.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:24.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:23 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:24.296 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:23 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/188884556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:24.297 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:24.297 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:24.297 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:24.297 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:24.297 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:24.297 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:23 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:24.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:23 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/188884556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:24.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:24.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:24.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:24.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:24.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:24.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:23 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:24.941 INFO:journalctl@ceph.mds.j.smithi046.stdout:Jun 23 01:07:24 smithi046 systemd[1]: Starting Ceph mds.j for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:24.942 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:24 smithi046 ceph-mon[84949]: Deploying daemon mds.j on smithi046 2023-06-23T01:07:24.942 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:24 smithi046 ceph-mon[84949]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:24.942 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:24 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.92:6832/1677883305,v1:172.21.15.92:6833/1677883305] up:boot 2023-06-23T01:07:24.942 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:24 smithi046 ceph-mon[84949]: fsmap 2 up:standby 2023-06-23T01:07:24.942 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:24 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-06-23T01:07:25.008 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:25.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.131+0000 7fd48edba700 1 -- 172.21.15.46:0/1277302386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880fcb00 msgr2=0x7fd4880fcf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:25.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.131+0000 7fd48edba700 1 --2- 172.21.15.46:0/1277302386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880fcb00 0x7fd4880fcf10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd4840049c0 tx=0x7fd484009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:25.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.131+0000 7fd48edba700 1 -- 172.21.15.46:0/1277302386 shutdown_connections 2023-06-23T01:07:25.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.131+0000 7fd48edba700 1 --2- 172.21.15.46:0/1277302386 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4880fe6a0 0x7fd488104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:25.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.131+0000 7fd48edba700 1 --2- 172.21.15.46:0/1277302386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880fcb00 0x7fd4880fcf10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:25.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.131+0000 7fd48edba700 1 --2- 172.21.15.46:0/1277302386 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd4880fdd10 0x7fd4880fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:25.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.131+0000 7fd48edba700 1 -- 172.21.15.46:0/1277302386 >> 172.21.15.46:0/1277302386 conn(0x7fd4880f8260 msgr2=0x7fd4880fa690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:25.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48edba700 1 -- 172.21.15.46:0/1277302386 shutdown_connections 2023-06-23T01:07:25.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48edba700 1 -- 172.21.15.46:0/1277302386 wait complete. 2023-06-23T01:07:25.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48edba700 1 Processor -- start 2023-06-23T01:07:25.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48edba700 1 -- start start 2023-06-23T01:07:25.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48edba700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880fcb00 0x7fd48810d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:25.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48edba700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd4880fdd10 0x7fd48810dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:25.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48edba700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4880fe6a0 0x7fd488114c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:25.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48edba700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4881152a0 con 0x7fd4880fcb00 2023-06-23T01:07:25.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48edba700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd488115410 con 0x7fd4880fe6a0 2023-06-23T01:07:25.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48edba700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4881155d0 con 0x7fd4880fdd10 2023-06-23T01:07:25.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48ddb8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880fcb00 0x7fd48810d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:25.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48ddb8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880fcb00 0x7fd48810d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59718/0 (socket says 172.21.15.46:59718) 2023-06-23T01:07:25.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48ddb8700 1 -- 172.21.15.46:0/1245400653 learned_addr learned my addr 172.21.15.46:0/1245400653 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48d5b7700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd4880fdd10 0x7fd48810dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.132+0000 7fd48e5b9700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4880fe6a0 0x7fd488114c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.133+0000 7fd48ddb8700 1 -- 172.21.15.46:0/1245400653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd4880fdd10 msgr2=0x7fd48810dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.133+0000 7fd48ddb8700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd4880fdd10 0x7fd48810dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.133+0000 7fd48ddb8700 1 -- 172.21.15.46:0/1245400653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4880fe6a0 msgr2=0x7fd488114c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.133+0000 7fd48ddb8700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4880fe6a0 0x7fd488114c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.133+0000 7fd48ddb8700 1 -- 172.21.15.46:0/1245400653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd484005040 con 0x7fd4880fcb00 2023-06-23T01:07:25.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.133+0000 7fd48ddb8700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880fcb00 0x7fd48810d6c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd48400b920 tx=0x7fd484009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.133+0000 7fd47e7fc700 1 -- 172.21.15.46:0/1245400653 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48400e040 con 0x7fd4880fcb00 2023-06-23T01:07:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.133+0000 7fd47e7fc700 1 -- 172.21.15.46:0/1245400653 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd484009570 con 0x7fd4880fcb00 2023-06-23T01:07:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.133+0000 7fd48edba700 1 -- 172.21.15.46:0/1245400653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd488115850 con 0x7fd4880fcb00 2023-06-23T01:07:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.133+0000 7fd47e7fc700 1 -- 172.21.15.46:0/1245400653 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd484013d20 con 0x7fd4880fcb00 2023-06-23T01:07:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.133+0000 7fd48edba700 1 -- 172.21.15.46:0/1245400653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd488115cb0 con 0x7fd4880fcb00 2023-06-23T01:07:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.134+0000 7fd48edba700 1 -- 172.21.15.46:0/1245400653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd488104070 con 0x7fd4880fcb00 2023-06-23T01:07:25.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.135+0000 7fd47e7fc700 1 -- 172.21.15.46:0/1245400653 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fd4840134b0 con 0x7fd4880fcb00 2023-06-23T01:07:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.136+0000 7fd47e7fc700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd47409b680 0x7fd47409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.136+0000 7fd48d5b7700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd47409b680 0x7fd47409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:25.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.136+0000 7fd47e7fc700 1 -- 172.21.15.46:0/1245400653 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fd4840c5b70 con 0x7fd4880fcb00 2023-06-23T01:07:25.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.136+0000 7fd48d5b7700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd47409b680 0x7fd47409db30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd478009b80 tx=0x7fd47800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:25.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.140+0000 7fd47e7fc700 1 -- 172.21.15.46:0/1245400653 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fd484091870 con 0x7fd4880fcb00 2023-06-23T01:07:25.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:24 smithi131 ceph-mon[86495]: Deploying daemon mds.j on smithi046 2023-06-23T01:07:25.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:24 smithi131 ceph-mon[86495]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:25.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:24 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.92:6832/1677883305,v1:172.21.15.92:6833/1677883305] up:boot 2023-06-23T01:07:25.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:24 smithi131 ceph-mon[86495]: fsmap 2 up:standby 2023-06-23T01:07:25.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:24 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-06-23T01:07:25.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:24 smithi092 ceph-mon[85670]: Deploying daemon mds.j on smithi046 2023-06-23T01:07:25.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:24 smithi092 ceph-mon[85670]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:25.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:24 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.92:6832/1677883305,v1:172.21.15.92:6833/1677883305] up:boot 2023-06-23T01:07:25.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:24 smithi092 ceph-mon[85670]: fsmap 2 up:standby 2023-06-23T01:07:25.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:24 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-06-23T01:07:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.394+0000 7fd48edba700 1 -- 172.21.15.46:0/1245400653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4881081a0 con 0x7fd4880fcb00 2023-06-23T01:07:25.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.394+0000 7fd47e7fc700 1 -- 172.21.15.46:0/1245400653 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2017 (secure 0 0 0) 0x7fd484094e90 con 0x7fd4880fcb00 2023-06-23T01:07:25.397 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:25.398 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/3255240190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":3255240190},{"type":"v1","addr":"172.21.15.131:6833","nonce":3255240190}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1677883305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1677883305},{"type":"v1","addr":"172.21.15.92:6833","nonce":1677883305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-06-23T01:07:25.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 -- 172.21.15.46:0/1245400653 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd47409b680 msgr2=0x7fd47409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:25.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd47409b680 0x7fd47409db30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd478009b80 tx=0x7fd47800b040 comp rx=0 tx=0).stop 2023-06-23T01:07:25.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 -- 172.21.15.46:0/1245400653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880fcb00 msgr2=0x7fd48810d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:25.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880fcb00 0x7fd48810d6c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd48400b920 tx=0x7fd484009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:25.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 -- 172.21.15.46:0/1245400653 shutdown_connections 2023-06-23T01:07:25.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd47409b680 0x7fd47409db30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:25.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd4880fe6a0 0x7fd488114c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:25.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4880fcb00 0x7fd48810d6c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:25.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 --2- 172.21.15.46:0/1245400653 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd4880fdd10 0x7fd48810dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:25.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 -- 172.21.15.46:0/1245400653 >> 172.21.15.46:0/1245400653 conn(0x7fd4880f8260 msgr2=0x7fd4881011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:25.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 -- 172.21.15.46:0/1245400653 shutdown_connections 2023-06-23T01:07:25.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:25.396+0000 7fd48edba700 1 -- 172.21.15.46:0/1245400653 wait complete. 2023-06-23T01:07:25.402 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 3 2023-06-23T01:07:25.818 INFO:journalctl@ceph.mds.j.smithi046.stdout:Jun 23 01:07:25 smithi046 bash[128514]: 9f392e69db0ac679121017acfac115e966cd4996dcad697de75300cb23f696d7 2023-06-23T01:07:25.818 INFO:journalctl@ceph.mds.j.smithi046.stdout:Jun 23 01:07:25 smithi046 ceph-mds[128618]: -- 172.21.15.46:0/392370495 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558f4e68c9c0 con 0x558f4e663000 2023-06-23T01:07:25.818 INFO:journalctl@ceph.mds.j.smithi046.stdout:Jun 23 01:07:25 smithi046 conmon[128594]: starting mds.j at 2023-06-23T01:07:25.818 INFO:journalctl@ceph.mds.j.smithi046.stdout:Jun 23 01:07:25 smithi046 systemd[1]: Started Ceph mds.j for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:26.547 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/1245400653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:26.547 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:26.547 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:26.547 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:26.548 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:26.548 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:26.548 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:26.548 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: Deploying daemon mds.d on smithi046 2023-06-23T01:07:26.548 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:26.548 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.46:6834/2158355110,v1:172.21.15.46:6835/2158355110] up:boot 2023-06-23T01:07:26.548 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: fsmap 3 up:standby 2023-06-23T01:07:26.548 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:26 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-06-23T01:07:26.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/1245400653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:26.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:26.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:26.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:26.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:26.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:26.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:26.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: Deploying daemon mds.d on smithi046 2023-06-23T01:07:26.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:26.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.46:6834/2158355110,v1:172.21.15.46:6835/2158355110] up:boot 2023-06-23T01:07:26.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: fsmap 3 up:standby 2023-06-23T01:07:26.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:26 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-06-23T01:07:26.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/1245400653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:26.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:26.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:26.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:26.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:26.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:26.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:26.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: Deploying daemon mds.d on smithi046 2023-06-23T01:07:26.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:26.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.46:6834/2158355110,v1:172.21.15.46:6835/2158355110] up:boot 2023-06-23T01:07:26.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: fsmap 3 up:standby 2023-06-23T01:07:26.734 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:26 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-06-23T01:07:27.319 INFO:journalctl@ceph.mds.d.smithi046.stdout:Jun 23 01:07:27 smithi046 systemd[1]: Starting Ceph mds.d for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:27.414 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.542+0000 7f622a1c7700 1 -- 172.21.15.46:0/1333903865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62240fce60 msgr2=0x7f62240fd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:27.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.542+0000 7f622a1c7700 1 --2- 172.21.15.46:0/1333903865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62240fce60 0x7f62240fd2d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f620c0049c0 tx=0x7f620c009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.542+0000 7f622a1c7700 1 -- 172.21.15.46:0/1333903865 shutdown_connections 2023-06-23T01:07:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.542+0000 7f622a1c7700 1 --2- 172.21.15.46:0/1333903865 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62240fd810 0x7f62241040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.542+0000 7f622a1c7700 1 --2- 172.21.15.46:0/1333903865 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62240fbbb0 0x7f62240fbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.542+0000 7f622a1c7700 1 --2- 172.21.15.46:0/1333903865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62240fce60 0x7f62240fd2d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.542+0000 7f622a1c7700 1 -- 172.21.15.46:0/1333903865 >> 172.21.15.46:0/1333903865 conn(0x7f62240f72d0 msgr2=0x7f62240f9740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.542+0000 7f622a1c7700 1 -- 172.21.15.46:0/1333903865 shutdown_connections 2023-06-23T01:07:27.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.542+0000 7f622a1c7700 1 -- 172.21.15.46:0/1333903865 wait complete. 2023-06-23T01:07:27.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.543+0000 7f622a1c7700 1 Processor -- start 2023-06-23T01:07:27.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.543+0000 7f622a1c7700 1 -- start start 2023-06-23T01:07:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.543+0000 7f622a1c7700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62240fbbb0 0x7f622410d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.543+0000 7f622a1c7700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62240fce60 0x7f622410dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.543+0000 7f622a1c7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62240fd810 0x7f6224114d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.543+0000 7f622a1c7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6224115390 con 0x7f62240fd810 2023-06-23T01:07:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.543+0000 7f622a1c7700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6224115500 con 0x7f62240fbbb0 2023-06-23T01:07:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.543+0000 7f622a1c7700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62241156c0 con 0x7f62240fce60 2023-06-23T01:07:27.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.543+0000 7f6223fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62240fd810 0x7f6224114d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.543+0000 7f6223fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62240fd810 0x7f6224114d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59752/0 (socket says 172.21.15.46:59752) 2023-06-23T01:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.543+0000 7f6223fff700 1 -- 172.21.15.46:0/2961093406 learned_addr learned my addr 172.21.15.46:0/2961093406 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f62237fe700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62240fbbb0 0x7f622410d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f621affd700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62240fce60 0x7f622410dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f6223fff700 1 -- 172.21.15.46:0/2961093406 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62240fce60 msgr2=0x7f622410dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f6223fff700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62240fce60 0x7f622410dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f6223fff700 1 -- 172.21.15.46:0/2961093406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62240fbbb0 msgr2=0x7f622410d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f6223fff700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62240fbbb0 0x7f622410d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f6223fff700 1 -- 172.21.15.46:0/2961093406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620c005040 con 0x7f62240fd810 2023-06-23T01:07:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f6223fff700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62240fd810 0x7f6224114d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f621c00cb90 tx=0x7f621c007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f6220ff9700 1 -- 172.21.15.46:0/2961093406 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c00a870 con 0x7f62240fd810 2023-06-23T01:07:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f6220ff9700 1 -- 172.21.15.46:0/2961093406 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f621c00ae70 con 0x7f62240fd810 2023-06-23T01:07:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f622a1c7700 1 -- 172.21.15.46:0/2961093406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62241159a0 con 0x7f62240fd810 2023-06-23T01:07:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f6220ff9700 1 -- 172.21.15.46:0/2961093406 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c012e80 con 0x7f62240fd810 2023-06-23T01:07:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.544+0000 7f622a1c7700 1 -- 172.21.15.46:0/2961093406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6224115f10 con 0x7f62240fd810 2023-06-23T01:07:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.545+0000 7f622a1c7700 1 -- 172.21.15.46:0/2961093406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6224103d60 con 0x7f62240fd810 2023-06-23T01:07:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.546+0000 7f6220ff9700 1 -- 172.21.15.46:0/2961093406 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f621c00a9d0 con 0x7f62240fd810 2023-06-23T01:07:27.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.547+0000 7f6220ff9700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f621009b680 0x7f621009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.547+0000 7f62237fe700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f621009b680 0x7f621009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.547+0000 7f6220ff9700 1 -- 172.21.15.46:0/2961093406 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f621c0c4f10 con 0x7f62240fd810 2023-06-23T01:07:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.547+0000 7f62237fe700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f621009b680 0x7f621009db30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6214009fd0 tx=0x7f621400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:27.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.551+0000 7f6220ff9700 1 -- 172.21.15.46:0/2961093406 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f621c090b70 con 0x7f62240fd810 2023-06-23T01:07:27.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.800+0000 7f622a1c7700 1 -- 172.21.15.46:0/2961093406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6224107220 con 0x7f62240fd810 2023-06-23T01:07:27.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.800+0000 7f6220ff9700 1 -- 172.21.15.46:0/2961093406 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2774 (secure 0 0 0) 0x7f621c094190 con 0x7f62240fd810 2023-06-23T01:07:27.804 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:27.804 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2158355110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2158355110},{"type":"v1","addr":"172.21.15.46:6835","nonce":2158355110}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/3255240190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":3255240190},{"type":"v1","addr":"172.21.15.131:6833","nonce":3255240190}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1677883305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1677883305},{"type":"v1","addr":"172.21.15.92:6833","nonce":1677883305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-06-23T01:07:27.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 -- 172.21.15.46:0/2961093406 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f621009b680 msgr2=0x7f621009db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:27.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f621009b680 0x7f621009db30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6214009fd0 tx=0x7f621400b040 comp rx=0 tx=0).stop 2023-06-23T01:07:27.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 -- 172.21.15.46:0/2961093406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62240fd810 msgr2=0x7f6224114d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62240fd810 0x7f6224114d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f621c00cb90 tx=0x7f621c007ed0 comp rx=0 tx=0).stop 2023-06-23T01:07:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 -- 172.21.15.46:0/2961093406 shutdown_connections 2023-06-23T01:07:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f621009b680 0x7f621009db30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:27.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f62240fbbb0 0x7f622410d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:27.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62240fd810 0x7f6224114d80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:27.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 --2- 172.21.15.46:0/2961093406 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f62240fce60 0x7f622410dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:27.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 -- 172.21.15.46:0/2961093406 >> 172.21.15.46:0/2961093406 conn(0x7f62240f72d0 msgr2=0x7f62240f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:27.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 -- 172.21.15.46:0/2961093406 shutdown_connections 2023-06-23T01:07:27.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:27.803+0000 7f622a1c7700 1 -- 172.21.15.46:0/2961093406 wait complete. 2023-06-23T01:07:27.810 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 4 2023-06-23T01:07:27.817 INFO:journalctl@ceph.mds.d.smithi046.stdout:Jun 23 01:07:27 smithi046 bash[129125]: 0af7bdd4cdfe775b0c54d57d4c5b0c2ea055af5d3721ce00d467114a475618e5 2023-06-23T01:07:27.817 INFO:journalctl@ceph.mds.d.smithi046.stdout:Jun 23 01:07:27 smithi046 ceph-mds[129229]: -- 172.21.15.46:0/111347420 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ae80a809c0 con 0x55ae80a57000 2023-06-23T01:07:27.817 INFO:journalctl@ceph.mds.d.smithi046.stdout:Jun 23 01:07:27 smithi046 conmon[129166]: starting mds.d at 2023-06-23T01:07:27.818 INFO:journalctl@ceph.mds.d.smithi046.stdout:Jun 23 01:07:27 smithi046 systemd[1]: Started Ceph mds.d for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:28.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:28.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:28.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:28.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:28.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:28.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:28.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: Deploying daemon mds.e on smithi092 2023-06-23T01:07:28.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2961093406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:28.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:28.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.46:6836/2977827695,v1:172.21.15.46:6837/2977827695] up:boot 2023-06-23T01:07:28.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: fsmap 4 up:standby 2023-06-23T01:07:28.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:28 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-06-23T01:07:29.062 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:29.062 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:29.063 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:29.063 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:29.063 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:29.063 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:29.063 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: Deploying daemon mds.e on smithi092 2023-06-23T01:07:29.063 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2961093406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:29.063 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:29.064 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.46:6836/2977827695,v1:172.21.15.46:6837/2977827695] up:boot 2023-06-23T01:07:29.064 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: fsmap 4 up:standby 2023-06-23T01:07:29.064 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:28 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-06-23T01:07:29.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:29.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:29.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:29.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:29.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:29.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:29.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: Deploying daemon mds.e on smithi092 2023-06-23T01:07:29.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2961093406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:29.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:29.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.46:6836/2977827695,v1:172.21.15.46:6837/2977827695] up:boot 2023-06-23T01:07:29.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: fsmap 4 up:standby 2023-06-23T01:07:29.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:28 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-06-23T01:07:29.391 INFO:journalctl@ceph.mds.e.smithi092.stdout:Jun 23 01:07:29 smithi092 systemd[1]: Starting Ceph mds.e for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:29.820 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:29.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.954+0000 7f513aa13700 1 -- 172.21.15.46:0/3409924165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5134066da0 msgr2=0x7f51340ff950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:29.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.954+0000 7f513aa13700 1 --2- 172.21.15.46:0/3409924165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5134066da0 0x7f51340ff950 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f511c0049c0 tx=0x7f511c009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:29.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.954+0000 7f513aa13700 1 -- 172.21.15.46:0/3409924165 shutdown_connections 2023-06-23T01:07:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.954+0000 7f513aa13700 1 --2- 172.21.15.46:0/3409924165 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5134102980 0x7f5134104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.954+0000 7f513aa13700 1 --2- 172.21.15.46:0/3409924165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5134066da0 0x7f51340ff950 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.954+0000 7f513aa13700 1 --2- 172.21.15.46:0/3409924165 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f51340ffe90 0x7f5134102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.954+0000 7f513aa13700 1 -- 172.21.15.46:0/3409924165 >> 172.21.15.46:0/3409924165 conn(0x7f51340f7300 msgr2=0x7f51340f9750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.954+0000 7f513aa13700 1 -- 172.21.15.46:0/3409924165 shutdown_connections 2023-06-23T01:07:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.954+0000 7f513aa13700 1 -- 172.21.15.46:0/3409924165 wait complete. 2023-06-23T01:07:29.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f513aa13700 1 Processor -- start 2023-06-23T01:07:29.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f513aa13700 1 -- start start 2023-06-23T01:07:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f513aa13700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5134066da0 0x7f513410d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f513aa13700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51340ffe90 0x7f513410de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f513aa13700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5134102980 0x7f5134114ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f513aa13700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5134115500 con 0x7f51340ffe90 2023-06-23T01:07:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f513aa13700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5134115670 con 0x7f5134066da0 2023-06-23T01:07:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f513aa13700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5134115830 con 0x7f5134102980 2023-06-23T01:07:29.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f51337fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51340ffe90 0x7f513410de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f51337fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51340ffe90 0x7f513410de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59782/0 (socket says 172.21.15.46:59782) 2023-06-23T01:07:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f51337fe700 1 -- 172.21.15.46:0/667034343 learned_addr learned my addr 172.21.15.46:0/667034343 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f5138fb0700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5134102980 0x7f5134114ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.955+0000 7f5133fff700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5134066da0 0x7f513410d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.956+0000 7f51337fe700 1 -- 172.21.15.46:0/667034343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5134102980 msgr2=0x7f5134114ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:29.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.956+0000 7f51337fe700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5134102980 0x7f5134114ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.956+0000 7f51337fe700 1 -- 172.21.15.46:0/667034343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5134066da0 msgr2=0x7f513410d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.956+0000 7f51337fe700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5134066da0 0x7f513410d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.956+0000 7f51337fe700 1 -- 172.21.15.46:0/667034343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c005040 con 0x7f51340ffe90 2023-06-23T01:07:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.956+0000 7f51337fe700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51340ffe90 0x7f513410de60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5124002ad0 tx=0x7f51240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.956+0000 7f5130ff9700 1 -- 172.21.15.46:0/667034343 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5124007cc0 con 0x7f51340ffe90 2023-06-23T01:07:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.956+0000 7f5130ff9700 1 -- 172.21.15.46:0/667034343 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f512400f040 con 0x7f51340ffe90 2023-06-23T01:07:29.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.956+0000 7f513aa13700 1 -- 172.21.15.46:0/667034343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5134115b10 con 0x7f51340ffe90 2023-06-23T01:07:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.956+0000 7f5130ff9700 1 -- 172.21.15.46:0/667034343 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512400d830 con 0x7f51340ffe90 2023-06-23T01:07:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.956+0000 7f513aa13700 1 -- 172.21.15.46:0/667034343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5134116000 con 0x7f51340ffe90 2023-06-23T01:07:29.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.957+0000 7f513aa13700 1 -- 172.21.15.46:0/667034343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51341083d0 con 0x7f51340ffe90 2023-06-23T01:07:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.959+0000 7f5130ff9700 1 -- 172.21.15.46:0/667034343 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f512400dc60 con 0x7f51340ffe90 2023-06-23T01:07:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.959+0000 7f5130ff9700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f512009b680 0x7f512009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:29.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.959+0000 7f5133fff700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f512009b680 0x7f512009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.959+0000 7f5130ff9700 1 -- 172.21.15.46:0/667034343 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f51240c4d00 con 0x7f51340ffe90 2023-06-23T01:07:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.960+0000 7f5133fff700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f512009b680 0x7f512009db30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f511c004560 tx=0x7f511c009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:29.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:29.963+0000 7f5130ff9700 1 -- 172.21.15.46:0/667034343 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f5124090960 con 0x7f51340ffe90 2023-06-23T01:07:30.139 INFO:journalctl@ceph.mds.e.smithi092.stdout:Jun 23 01:07:29 smithi092 ceph-mds[117165]: -- 172.21.15.92:0/780780323 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f42687a9c0 con 0x55f426850800 2023-06-23T01:07:30.140 INFO:journalctl@ceph.mds.e.smithi092.stdout:Jun 23 01:07:29 smithi092 conmon[117140]: starting mds.e at 2023-06-23T01:07:30.140 INFO:journalctl@ceph.mds.e.smithi092.stdout:Jun 23 01:07:29 smithi092 bash[117100]: 843018508e6a02369c89e5782339fcfe5d53eb04646a6dcf3617c2895cf0b426 2023-06-23T01:07:30.140 INFO:journalctl@ceph.mds.e.smithi092.stdout:Jun 23 01:07:29 smithi092 systemd[1]: Started Ceph mds.e for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.202+0000 7f513aa13700 1 -- 172.21.15.46:0/667034343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51341161f0 con 0x7f51340ffe90 2023-06-23T01:07:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.203+0000 7f5130ff9700 1 -- 172.21.15.46:0/667034343 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4288 (secure 0 0 0) 0x7f5124093f80 con 0x7f51340ffe90 2023-06-23T01:07:30.206 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:30.206 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2158355110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2158355110},{"type":"v1","addr":"172.21.15.46:6835","nonce":2158355110}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2977827695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2977827695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2977827695}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/3255240190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":3255240190},{"type":"v1","addr":"172.21.15.131:6833","nonce":3255240190}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1677883305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1677883305},{"type":"v1","addr":"172.21.15.92:6833","nonce":1677883305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/2078557791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2078557791},{"type":"v1","addr":"172.21.15.92:6835","nonce":2078557791}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-06-23T01:07:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 -- 172.21.15.46:0/667034343 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f512009b680 msgr2=0x7f512009db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f512009b680 0x7f512009db30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f511c004560 tx=0x7f511c009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 -- 172.21.15.46:0/667034343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51340ffe90 msgr2=0x7f513410de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51340ffe90 0x7f513410de60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5124002ad0 tx=0x7f51240046b0 comp rx=0 tx=0).stop 2023-06-23T01:07:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 -- 172.21.15.46:0/667034343 shutdown_connections 2023-06-23T01:07:30.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f512009b680 0x7f512009db30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5134066da0 0x7f513410d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f5134102980 0x7f5134114ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 --2- 172.21.15.46:0/667034343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51340ffe90 0x7f513410de60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 -- 172.21.15.46:0/667034343 >> 172.21.15.46:0/667034343 conn(0x7f51340f7300 msgr2=0x7f51340fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 -- 172.21.15.46:0/667034343 shutdown_connections 2023-06-23T01:07:30.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:30.205+0000 7f513aa13700 1 -- 172.21.15.46:0/667034343 wait complete. 2023-06-23T01:07:30.210 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2023-06-23T01:07:31.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.92:6834/2078557791,v1:172.21.15.92:6835/2078557791] up:boot 2023-06-23T01:07:31.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: fsmap 5 up:standby 2023-06-23T01:07:31.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-06-23T01:07:31.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:31.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:31.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:31.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:31.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:31.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:31.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: Deploying daemon mds.f on smithi131 2023-06-23T01:07:31.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:31.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:30 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/667034343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:31.106 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.92:6834/2078557791,v1:172.21.15.92:6835/2078557791] up:boot 2023-06-23T01:07:31.107 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: fsmap 5 up:standby 2023-06-23T01:07:31.107 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-06-23T01:07:31.107 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:31.107 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:31.107 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:31.107 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:31.107 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:31.108 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:31.108 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: Deploying daemon mds.f on smithi131 2023-06-23T01:07:31.108 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:31.108 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:30 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/667034343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:31.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.92:6834/2078557791,v1:172.21.15.92:6835/2078557791] up:boot 2023-06-23T01:07:31.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: fsmap 5 up:standby 2023-06-23T01:07:31.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-06-23T01:07:31.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:31.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:31.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:31.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:31.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:31.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:31.142 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: Deploying daemon mds.f on smithi131 2023-06-23T01:07:31.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:31.143 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:30 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/667034343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:31.732 INFO:journalctl@ceph.mds.f.smithi131.stdout:Jun 23 01:07:31 smithi131 systemd[1]: Starting Ceph mds.f for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:32.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:32.231 INFO:journalctl@ceph.mds.f.smithi131.stdout:Jun 23 01:07:31 smithi131 ceph-mds[119587]: -- 172.21.15.131:0/3624758205 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55743342c9c0 con 0x557433402c00 2023-06-23T01:07:32.232 INFO:journalctl@ceph.mds.f.smithi131.stdout:Jun 23 01:07:31 smithi131 conmon[119563]: starting mds.f at 2023-06-23T01:07:32.232 INFO:journalctl@ceph.mds.f.smithi131.stdout:Jun 23 01:07:31 smithi131 bash[119523]: 53a63d8c04f840e6e77c96bad0580bb91f0393b2da3cd1da9ba5285f51f703c1 2023-06-23T01:07:32.232 INFO:journalctl@ceph.mds.f.smithi131.stdout:Jun 23 01:07:32 smithi131 systemd[1]: Started Ceph mds.f for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.353+0000 7fd306df8700 1 -- 172.21.15.46:0/3329074132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3000fc960 msgr2=0x7fd300103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.353+0000 7fd306df8700 1 --2- 172.21.15.46:0/3329074132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3000fc960 0x7fd300103200 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009d60 tx=0x7fd2f8004d10 comp rx=0 tx=0).stop 2023-06-23T01:07:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.353+0000 7fd306df8700 1 -- 172.21.15.46:0/3329074132 shutdown_connections 2023-06-23T01:07:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.353+0000 7fd306df8700 1 --2- 172.21.15.46:0/3329074132 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3000fad00 0x7fd3000fb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.353+0000 7fd306df8700 1 --2- 172.21.15.46:0/3329074132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3000fc960 0x7fd300103200 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.353+0000 7fd306df8700 1 --2- 172.21.15.46:0/3329074132 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3000fbfb0 0x7fd3000fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.353+0000 7fd306df8700 1 -- 172.21.15.46:0/3329074132 >> 172.21.15.46:0/3329074132 conn(0x7fd3000f6480 msgr2=0x7fd3000f8890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.353+0000 7fd306df8700 1 -- 172.21.15.46:0/3329074132 shutdown_connections 2023-06-23T01:07:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.353+0000 7fd306df8700 1 -- 172.21.15.46:0/3329074132 wait complete. 2023-06-23T01:07:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd306df8700 1 Processor -- start 2023-06-23T01:07:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd306df8700 1 -- start start 2023-06-23T01:07:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd306df8700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3000fad00 0x7fd300106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd306df8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3000fbfb0 0x7fd300105200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd306df8700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3000fc960 0x7fd300105740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd306df8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd300105eb0 con 0x7fd3000fbfb0 2023-06-23T01:07:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd306df8700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd300106020 con 0x7fd3000fc960 2023-06-23T01:07:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd306df8700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd300117680 con 0x7fd3000fad00 2023-06-23T01:07:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd3055f5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3000fbfb0 0x7fd300105200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd3055f5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3000fbfb0 0x7fd300105200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59792/0 (socket says 172.21.15.46:59792) 2023-06-23T01:07:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd3055f5700 1 -- 172.21.15.46:0/3885553915 learned_addr learned my addr 172.21.15.46:0/3885553915 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd305df6700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3000fad00 0x7fd300106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd3065f7700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3000fc960 0x7fd300105740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd3055f5700 1 -- 172.21.15.46:0/3885553915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3000fad00 msgr2=0x7fd300106ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd3055f5700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3000fad00 0x7fd300106ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd3055f5700 1 -- 172.21.15.46:0/3885553915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3000fc960 msgr2=0x7fd300105740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd3055f5700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3000fc960 0x7fd300105740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd3055f5700 1 -- 172.21.15.46:0/3885553915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f8009940 con 0x7fd3000fbfb0 2023-06-23T01:07:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd3055f5700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3000fbfb0 0x7fd300105200 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000ebc0 tx=0x7fd2f000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.354+0000 7fd2f67fc700 1 -- 172.21.15.46:0/3885553915 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0007cc0 con 0x7fd3000fbfb0 2023-06-23T01:07:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.355+0000 7fd306df8700 1 -- 172.21.15.46:0/3885553915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3001178d0 con 0x7fd3000fbfb0 2023-06-23T01:07:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.355+0000 7fd2f67fc700 1 -- 172.21.15.46:0/3885553915 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd2f000f040 con 0x7fd3000fbfb0 2023-06-23T01:07:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.355+0000 7fd306df8700 1 -- 172.21.15.46:0/3885553915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd300117e90 con 0x7fd3000fbfb0 2023-06-23T01:07:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.355+0000 7fd2f67fc700 1 -- 172.21.15.46:0/3885553915 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f000c820 con 0x7fd3000fbfb0 2023-06-23T01:07:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.355+0000 7fd306df8700 1 -- 172.21.15.46:0/3885553915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd300108210 con 0x7fd3000fbfb0 2023-06-23T01:07:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.357+0000 7fd2f67fc700 1 -- 172.21.15.46:0/3885553915 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fd2f000cc50 con 0x7fd3000fbfb0 2023-06-23T01:07:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.358+0000 7fd2f67fc700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd2ec09b680 0x7fd2ec09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.358+0000 7fd305df6700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd2ec09b680 0x7fd2ec09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.358+0000 7fd2f67fc700 1 -- 172.21.15.46:0/3885553915 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fd2f00c45d0 con 0x7fd3000fbfb0 2023-06-23T01:07:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.358+0000 7fd305df6700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd2ec09b680 0x7fd2ec09db30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009b80 tx=0x7fd2fc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.361+0000 7fd2f67fc700 1 -- 172.21.15.46:0/3885553915 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fd2f004da00 con 0x7fd3000fbfb0 2023-06-23T01:07:32.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.601+0000 7fd306df8700 1 -- 172.21.15.46:0/3885553915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3001083a0 con 0x7fd3000fbfb0 2023-06-23T01:07:32.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.601+0000 7fd2f67fc700 1 -- 172.21.15.46:0/3885553915 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4288 (secure 0 0 0) 0x7fd3001083a0 con 0x7fd3000fbfb0 2023-06-23T01:07:32.604 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:32.605 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2158355110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2158355110},{"type":"v1","addr":"172.21.15.46:6835","nonce":2158355110}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2977827695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2977827695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2977827695}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/3255240190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":3255240190},{"type":"v1","addr":"172.21.15.131:6833","nonce":3255240190}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1677883305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1677883305},{"type":"v1","addr":"172.21.15.92:6833","nonce":1677883305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/2078557791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2078557791},{"type":"v1","addr":"172.21.15.92:6835","nonce":2078557791}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-06-23T01:07:32.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 -- 172.21.15.46:0/3885553915 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd2ec09b680 msgr2=0x7fd2ec09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:32.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd2ec09b680 0x7fd2ec09db30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc009b80 tx=0x7fd2fc00b040 comp rx=0 tx=0).stop 2023-06-23T01:07:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 -- 172.21.15.46:0/3885553915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3000fbfb0 msgr2=0x7fd300105200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3000fbfb0 0x7fd300105200 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000ebc0 tx=0x7fd2f000b040 comp rx=0 tx=0).stop 2023-06-23T01:07:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 -- 172.21.15.46:0/3885553915 shutdown_connections 2023-06-23T01:07:32.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fd2ec09b680 0x7fd2ec09db30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:32.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd3000fc960 0x7fd300105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:32.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fd3000fad00 0x7fd300106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:32.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 --2- 172.21.15.46:0/3885553915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3000fbfb0 0x7fd300105200 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:32.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 -- 172.21.15.46:0/3885553915 >> 172.21.15.46:0/3885553915 conn(0x7fd3000f6480 msgr2=0x7fd3000f8560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:32.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 -- 172.21.15.46:0/3885553915 shutdown_connections 2023-06-23T01:07:32.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:32.603+0000 7fd306df8700 1 -- 172.21.15.46:0/3885553915 wait complete. 2023-06-23T01:07:32.610 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2023-06-23T01:07:33.164 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:32 smithi131 ceph-mon[86495]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:33.164 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:32 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:33.164 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:32 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:33.164 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:32 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:33.165 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:32 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:33.165 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:32 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:33.165 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:32 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:33.165 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:32 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/3885553915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:33.165 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:32 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.131:6834/653916810,v1:172.21.15.131:6835/653916810] up:boot 2023-06-23T01:07:33.165 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:32 smithi131 ceph-mon[86495]: fsmap 6 up:standby 2023-06-23T01:07:33.165 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:32 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-06-23T01:07:33.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:32 smithi046 ceph-mon[84949]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:33.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:32 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:33.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:32 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:33.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:32 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:33.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:32 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:33.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:32 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:33.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:32 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:33.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:32 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3885553915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:33.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:32 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.131:6834/653916810,v1:172.21.15.131:6835/653916810] up:boot 2023-06-23T01:07:33.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:32 smithi046 ceph-mon[84949]: fsmap 6 up:standby 2023-06-23T01:07:33.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:32 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-06-23T01:07:33.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:32 smithi092 ceph-mon[85670]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:33.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:32 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:33.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:32 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:33.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:32 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:33.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:32 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:33.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:32 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:33.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:32 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:33.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:32 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/3885553915' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:33.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:32 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.131:6834/653916810,v1:172.21.15.131:6835/653916810] up:boot 2023-06-23T01:07:33.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:32 smithi092 ceph-mon[85670]: fsmap 6 up:standby 2023-06-23T01:07:33.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:32 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-06-23T01:07:33.973 INFO:journalctl@ceph.mds.c.smithi131.stdout:Jun 23 01:07:33 smithi131 systemd[1]: Starting Ceph mds.c for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:34.227 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:33 smithi131 ceph-mon[86495]: Deploying daemon mds.c on smithi131 2023-06-23T01:07:34.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:33 smithi046 ceph-mon[84949]: Deploying daemon mds.c on smithi131 2023-06-23T01:07:34.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:33 smithi092 ceph-mon[85670]: Deploying daemon mds.c on smithi131 2023-06-23T01:07:34.482 INFO:journalctl@ceph.mds.c.smithi131.stdout:Jun 23 01:07:34 smithi131 ceph-mds[120149]: -- 172.21.15.131:0/3190688025 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56280b1fa9c0 con 0x56280b1d1000 2023-06-23T01:07:34.482 INFO:journalctl@ceph.mds.c.smithi131.stdout:Jun 23 01:07:34 smithi131 conmon[120125]: starting mds.c at 2023-06-23T01:07:34.482 INFO:journalctl@ceph.mds.c.smithi131.stdout:Jun 23 01:07:34 smithi131 bash[120085]: 5e8d31d93cca7ca8aff7d0a4852fdad2ffb5907c7e6dabf40eadc859854b6aa8 2023-06-23T01:07:34.482 INFO:journalctl@ceph.mds.c.smithi131.stdout:Jun 23 01:07:34 smithi131 systemd[1]: Started Ceph mds.c for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:34.619 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:34.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.750+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/297826243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef00fd770 msgr2=0x7f1ef0104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:34.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.750+0000 7f1ef7ee1700 1 --2- 172.21.15.46:0/297826243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef00fd770 0x7f1ef0104030 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8009d60 tx=0x7f1ee8004d10 comp rx=0 tx=0).stop 2023-06-23T01:07:34.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.750+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/297826243 shutdown_connections 2023-06-23T01:07:34.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.750+0000 7f1ef7ee1700 1 --2- 172.21.15.46:0/297826243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef00fd770 0x7f1ef0104030 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:34.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.750+0000 7f1ef7ee1700 1 --2- 172.21.15.46:0/297826243 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1ef00fbbd0 0x7f1ef00fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:34.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.750+0000 7f1ef7ee1700 1 --2- 172.21.15.46:0/297826243 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ef00fcde0 0x7f1ef00fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:34.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.750+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/297826243 >> 172.21.15.46:0/297826243 conn(0x7f1ef00f7310 msgr2=0x7f1ef00f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:34.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.750+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/297826243 shutdown_connections 2023-06-23T01:07:34.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.750+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/297826243 wait complete. 2023-06-23T01:07:34.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef7ee1700 1 Processor -- start 2023-06-23T01:07:34.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef7ee1700 1 -- start start 2023-06-23T01:07:34.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef7ee1700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1ef00fbbd0 0x7f1ef010d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:34.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef7ee1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef00fcde0 0x7f1ef010db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:34.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef7ee1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ef00fd770 0x7f1ef0114b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:34.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef7ee1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef01151a0 con 0x7f1ef00fcde0 2023-06-23T01:07:34.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef7ee1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef0115310 con 0x7f1ef00fd770 2023-06-23T01:07:34.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef7ee1700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef01154d0 con 0x7f1ef00fbbd0 2023-06-23T01:07:34.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef547c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef00fcde0 0x7f1ef010db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:34.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef547c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef00fcde0 0x7f1ef010db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54398/0 (socket says 172.21.15.46:54398) 2023-06-23T01:07:34.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef547c700 1 -- 172.21.15.46:0/2250818876 learned_addr learned my addr 172.21.15.46:0/2250818876 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:34.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef647e700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ef00fd770 0x7f1ef0114b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:34.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef5c7d700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1ef00fbbd0 0x7f1ef010d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:34.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef547c700 1 -- 172.21.15.46:0/2250818876 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1ef00fbbd0 msgr2=0x7f1ef010d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:34.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef547c700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1ef00fbbd0 0x7f1ef010d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:34.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef547c700 1 -- 172.21.15.46:0/2250818876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ef00fd770 msgr2=0x7f1ef0114b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:34.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef547c700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ef00fd770 0x7f1ef0114b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:34.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef547c700 1 -- 172.21.15.46:0/2250818876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee8009940 con 0x7f1ef00fcde0 2023-06-23T01:07:34.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.751+0000 7f1ef5c7d700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1ef00fbbd0 0x7f1ef010d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:07:34.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.752+0000 7f1ef547c700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef00fcde0 0x7f1ef010db00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0002ad0 tx=0x7f1ee00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:34.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.752+0000 7f1ee67fc700 1 -- 172.21.15.46:0/2250818876 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee0007cc0 con 0x7f1ef00fcde0 2023-06-23T01:07:34.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.752+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/2250818876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef01157b0 con 0x7f1ef00fcde0 2023-06-23T01:07:34.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.752+0000 7f1ee67fc700 1 -- 172.21.15.46:0/2250818876 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1ee000f040 con 0x7f1ef00fcde0 2023-06-23T01:07:34.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.752+0000 7f1ee67fc700 1 -- 172.21.15.46:0/2250818876 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee000d880 con 0x7f1ef00fcde0 2023-06-23T01:07:34.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.752+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/2250818876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef0115ca0 con 0x7f1ef00fcde0 2023-06-23T01:07:34.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.753+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/2250818876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef0103a10 con 0x7f1ef00fcde0 2023-06-23T01:07:34.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.754+0000 7f1ee67fc700 1 -- 172.21.15.46:0/2250818876 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f1ee000dcb0 con 0x7f1ef00fcde0 2023-06-23T01:07:34.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.754+0000 7f1ee67fc700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1edc09b680 0x7f1edc09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:34.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.754+0000 7f1ee67fc700 1 -- 172.21.15.46:0/2250818876 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f1ee00c4da0 con 0x7f1ef00fcde0 2023-06-23T01:07:34.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.754+0000 7f1ef5c7d700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1edc09b680 0x7f1edc09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:34.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.755+0000 7f1ef5c7d700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1edc09b680 0x7f1edc09db30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00fcc40 tx=0x7f1eec007ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:34.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.758+0000 7f1ee67fc700 1 -- 172.21.15.46:0/2250818876 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f1ee0090aa0 con 0x7f1ef00fcde0 2023-06-23T01:07:35.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.998+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/2250818876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ef0107370 con 0x7f1ef00fcde0 2023-06-23T01:07:35.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:34.999+0000 7f1ee67fc700 1 -- 172.21.15.46:0/2250818876 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5805 (secure 0 0 0) 0x7f1ee00940c0 con 0x7f1ef00fcde0 2023-06-23T01:07:35.002 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:35.002 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2158355110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2158355110},{"type":"v1","addr":"172.21.15.46:6835","nonce":2158355110}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2977827695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2977827695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2977827695}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/3255240190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":3255240190},{"type":"v1","addr":"172.21.15.131:6833","nonce":3255240190}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1677883305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1677883305},{"type":"v1","addr":"172.21.15.92:6833","nonce":1677883305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/653916810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":653916810},{"type":"v1","addr":"172.21.15.131:6835","nonce":653916810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/2078557791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2078557791},{"type":"v1","addr":"172.21.15.92:6835","nonce":2078557791}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2849145797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2849145797},{"type":"v1","addr":"172.21.15.131:6837","nonce":2849145797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-06-23T01:07:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/2250818876 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1edc09b680 msgr2=0x7f1edc09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1edc09b680 0x7f1edc09db30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1ef00fcc40 tx=0x7f1eec007ec0 comp rx=0 tx=0).stop 2023-06-23T01:07:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/2250818876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef00fcde0 msgr2=0x7f1ef010db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef00fcde0 0x7f1ef010db00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0002ad0 tx=0x7f1ee00046b0 comp rx=0 tx=0).stop 2023-06-23T01:07:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/2250818876 shutdown_connections 2023-06-23T01:07:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1edc09b680 0x7f1edc09db30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1ef00fd770 0x7f1ef0114b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1ef00fbbd0 0x7f1ef010d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 --2- 172.21.15.46:0/2250818876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ef00fcde0 0x7f1ef010db00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/2250818876 >> 172.21.15.46:0/2250818876 conn(0x7f1ef00f7310 msgr2=0x7f1ef00f92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/2250818876 shutdown_connections 2023-06-23T01:07:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:35.001+0000 7f1ef7ee1700 1 -- 172.21.15.46:0/2250818876 wait complete. 2023-06-23T01:07:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2023-06-23T01:07:35.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:34 smithi092 ceph-mon[85670]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:35.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:35.227 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:35.228 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:35.228 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:35.228 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:35.229 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:34 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:35.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:34 smithi131 ceph-mon[86495]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:35.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:35.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:35.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:35.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:35.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:35.234 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:34 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:35.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:34 smithi046 ceph-mon[84949]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:35.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:35.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:35.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:35.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:35.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:35.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:34 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:36.140 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:35 smithi092 ceph-mon[85670]: Deploying daemon mds.h on smithi092 2023-06-23T01:07:36.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:35 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.131:6836/2849145797,v1:172.21.15.131:6837/2849145797] up:boot 2023-06-23T01:07:36.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:35 smithi092 ceph-mon[85670]: fsmap 7 up:standby 2023-06-23T01:07:36.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:35 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-06-23T01:07:36.141 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:35 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2250818876' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:36.141 INFO:journalctl@ceph.mds.h.smithi092.stdout:Jun 23 01:07:35 smithi092 systemd[1]: Starting Ceph mds.h for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:36.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:35 smithi046 ceph-mon[84949]: Deploying daemon mds.h on smithi092 2023-06-23T01:07:36.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:35 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.131:6836/2849145797,v1:172.21.15.131:6837/2849145797] up:boot 2023-06-23T01:07:36.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:35 smithi046 ceph-mon[84949]: fsmap 7 up:standby 2023-06-23T01:07:36.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:35 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-06-23T01:07:36.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:35 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2250818876' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:36.481 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:35 smithi131 ceph-mon[86495]: Deploying daemon mds.h on smithi092 2023-06-23T01:07:36.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:35 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.131:6836/2849145797,v1:172.21.15.131:6837/2849145797] up:boot 2023-06-23T01:07:36.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:35 smithi131 ceph-mon[86495]: fsmap 7 up:standby 2023-06-23T01:07:36.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:35 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-06-23T01:07:36.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:35 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2250818876' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:36.639 INFO:journalctl@ceph.mds.h.smithi092.stdout:Jun 23 01:07:36 smithi092 ceph-mds[117722]: -- 172.21.15.92:0/4042315853 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d843f489c0 con 0x55d843f1f000 2023-06-23T01:07:36.640 INFO:journalctl@ceph.mds.h.smithi092.stdout:Jun 23 01:07:36 smithi092 conmon[117697]: starting mds.h at 2023-06-23T01:07:36.640 INFO:journalctl@ceph.mds.h.smithi092.stdout:Jun 23 01:07:36 smithi092 bash[117657]: ca2e6fe4a662d79b0b68dfd196b09e979210866205edec258b8d9dd8d602393c 2023-06-23T01:07:36.640 INFO:journalctl@ceph.mds.h.smithi092.stdout:Jun 23 01:07:36 smithi092 systemd[1]: Started Ceph mds.h for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:37.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 -- 172.21.15.46:0/1073370315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b80fe150 msgr2=0x7fb8b80fe560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 --2- 172.21.15.46:0/1073370315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b80fe150 0x7fb8b80fe560 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80049c0 tx=0x7fb8a8009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 -- 172.21.15.46:0/1073370315 shutdown_connections 2023-06-23T01:07:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 --2- 172.21.15.46:0/1073370315 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b80fb310 0x7fb8b8105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 --2- 172.21.15.46:0/1073370315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b80fe150 0x7fb8b80fe560 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 --2- 172.21.15.46:0/1073370315 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b80feaa0 0x7fb8b80fabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 -- 172.21.15.46:0/1073370315 >> 172.21.15.46:0/1073370315 conn(0x7fb8b80f68f0 msgr2=0x7fb8b80f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 -- 172.21.15.46:0/1073370315 shutdown_connections 2023-06-23T01:07:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 -- 172.21.15.46:0/1073370315 wait complete. 2023-06-23T01:07:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 Processor -- start 2023-06-23T01:07:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 -- start start 2023-06-23T01:07:37.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b80fb310 0x7fb8b810d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:37.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.146+0000 7fb8bed51700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b80fe150 0x7fb8b810dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:37.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8bed51700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b80feaa0 0x7fb8b8114c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:37.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8bed51700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b8115240 con 0x7fb8b80feaa0 2023-06-23T01:07:37.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8bed51700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b81153b0 con 0x7fb8b80fb310 2023-06-23T01:07:37.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8bed51700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b8115570 con 0x7fb8b80fe150 2023-06-23T01:07:37.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8be550700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b80feaa0 0x7fb8b8114c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:37.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8be550700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b80feaa0 0x7fb8b8114c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54412/0 (socket says 172.21.15.46:54412) 2023-06-23T01:07:37.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8be550700 1 -- 172.21.15.46:0/2505098468 learned_addr learned my addr 172.21.15.46:0/2505098468 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:37.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8bdd4f700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b80fb310 0x7fb8b810d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:37.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8bd54e700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b80fe150 0x7fb8b810dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:37.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8be550700 1 -- 172.21.15.46:0/2505098468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b80fe150 msgr2=0x7fb8b810dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:37.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8be550700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b80fe150 0x7fb8b810dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:37.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8be550700 1 -- 172.21.15.46:0/2505098468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b80fb310 msgr2=0x7fb8b810d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:37.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8be550700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b80fb310 0x7fb8b810d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8be550700 1 -- 172.21.15.46:0/2505098468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a8005040 con 0x7fb8b80feaa0 2023-06-23T01:07:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8bd54e700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b80fe150 0x7fb8b810dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:07:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8be550700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b80feaa0 0x7fb8b8114c30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000ec30 tx=0x7fb8b000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8ae7fc700 1 -- 172.21.15.46:0/2505098468 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0007b30 con 0x7fb8b80feaa0 2023-06-23T01:07:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8bed51700 1 -- 172.21.15.46:0/2505098468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b8115850 con 0x7fb8b80feaa0 2023-06-23T01:07:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8ae7fc700 1 -- 172.21.15.46:0/2505098468 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb8b000f040 con 0x7fb8b80feaa0 2023-06-23T01:07:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8ae7fc700 1 -- 172.21.15.46:0/2505098468 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b000ca20 con 0x7fb8b80feaa0 2023-06-23T01:07:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.147+0000 7fb8bed51700 1 -- 172.21.15.46:0/2505098468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b8115e10 con 0x7fb8b80feaa0 2023-06-23T01:07:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.148+0000 7fb8bed51700 1 -- 172.21.15.46:0/2505098468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b8103bb0 con 0x7fb8b80feaa0 2023-06-23T01:07:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.149+0000 7fb8ae7fc700 1 -- 172.21.15.46:0/2505098468 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fb8b001d400 con 0x7fb8b80feaa0 2023-06-23T01:07:37.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.150+0000 7fb8ae7fc700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb8a409b680 0x7fb8a409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:37.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.150+0000 7fb8bdd4f700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb8a409b680 0x7fb8a409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:37.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.150+0000 7fb8ae7fc700 1 -- 172.21.15.46:0/2505098468 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fb8b00c4e40 con 0x7fb8b80feaa0 2023-06-23T01:07:37.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.150+0000 7fb8bdd4f700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb8a409b680 0x7fb8a409db30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009b80 tx=0x7fb8a8009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:37.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.154+0000 7fb8ae7fc700 1 -- 172.21.15.46:0/2505098468 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fb8b0090b40 con 0x7fb8b80feaa0 2023-06-23T01:07:37.250 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:36 smithi131 ceph-mon[86495]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:37.250 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:37.250 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:37.251 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:37.251 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:37.251 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:37.251 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:36 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:37.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:36 smithi046 ceph-mon[84949]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:37.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:37.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:37.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:37.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:37.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:37.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:36 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:37.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:36 smithi092 ceph-mon[85670]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:37.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:37.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:37.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:37.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:37.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:37.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:36 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.398+0000 7fb8bed51700 1 -- 172.21.15.46:0/2505098468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8b802b210 con 0x7fb8b80feaa0 2023-06-23T01:07:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.399+0000 7fb8ae7fc700 1 -- 172.21.15.46:0/2505098468 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6559 (secure 0 0 0) 0x7fb8b0094160 con 0x7fb8b80feaa0 2023-06-23T01:07:37.402 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:37.402 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2158355110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2158355110},{"type":"v1","addr":"172.21.15.46:6835","nonce":2158355110}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2977827695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2977827695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2977827695}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/3255240190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":3255240190},{"type":"v1","addr":"172.21.15.131:6833","nonce":3255240190}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1677883305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1677883305},{"type":"v1","addr":"172.21.15.92:6833","nonce":1677883305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/653916810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":653916810},{"type":"v1","addr":"172.21.15.131:6835","nonce":653916810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/2078557791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2078557791},{"type":"v1","addr":"172.21.15.92:6835","nonce":2078557791}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2849145797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2849145797},{"type":"v1","addr":"172.21.15.131:6837","nonce":2849145797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/447743959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":447743959},{"type":"v1","addr":"172.21.15.92:6837","nonce":447743959}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-06-23T01:07:37.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 -- 172.21.15.46:0/2505098468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb8a409b680 msgr2=0x7fb8a409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb8a409b680 0x7fb8a409db30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009b80 tx=0x7fb8a8009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 -- 172.21.15.46:0/2505098468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b80feaa0 msgr2=0x7fb8b8114c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b80feaa0 0x7fb8b8114c30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000ec30 tx=0x7fb8b000b040 comp rx=0 tx=0).stop 2023-06-23T01:07:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 -- 172.21.15.46:0/2505098468 shutdown_connections 2023-06-23T01:07:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb8a409b680 0x7fb8a409db30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb8b80fb310 0x7fb8b810d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8b80feaa0 0x7fb8b8114c30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 --2- 172.21.15.46:0/2505098468 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb8b80fe150 0x7fb8b810dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 -- 172.21.15.46:0/2505098468 >> 172.21.15.46:0/2505098468 conn(0x7fb8b80f68f0 msgr2=0x7fb8b80f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 -- 172.21.15.46:0/2505098468 shutdown_connections 2023-06-23T01:07:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:37.401+0000 7fb8bed51700 1 -- 172.21.15.46:0/2505098468 wait complete. 2023-06-23T01:07:37.406 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2023-06-23T01:07:38.232 INFO:journalctl@ceph.mds.i.smithi131.stdout:Jun 23 01:07:38 smithi131 systemd[1]: Starting Ceph mds.i for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:38.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:37 smithi131 ceph-mon[86495]: Deploying daemon mds.i on smithi131 2023-06-23T01:07:38.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:37 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.92:6836/447743959,v1:172.21.15.92:6837/447743959] up:boot 2023-06-23T01:07:38.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:37 smithi131 ceph-mon[86495]: fsmap 8 up:standby 2023-06-23T01:07:38.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:37 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-06-23T01:07:38.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:37 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2505098468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:38.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:37 smithi046 ceph-mon[84949]: Deploying daemon mds.i on smithi131 2023-06-23T01:07:38.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:37 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.92:6836/447743959,v1:172.21.15.92:6837/447743959] up:boot 2023-06-23T01:07:38.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:37 smithi046 ceph-mon[84949]: fsmap 8 up:standby 2023-06-23T01:07:38.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:37 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-06-23T01:07:38.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:37 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2505098468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:38.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:37 smithi092 ceph-mon[85670]: Deploying daemon mds.i on smithi131 2023-06-23T01:07:38.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:37 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.92:6836/447743959,v1:172.21.15.92:6837/447743959] up:boot 2023-06-23T01:07:38.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:37 smithi092 ceph-mon[85670]: fsmap 8 up:standby 2023-06-23T01:07:38.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:37 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-06-23T01:07:38.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:37 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2505098468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:38.982 INFO:journalctl@ceph.mds.i.smithi131.stdout:Jun 23 01:07:38 smithi131 ceph-mds[120710]: -- 172.21.15.131:0/3866019488 <== mon.1 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559df806c9c0 con 0x559df8042800 2023-06-23T01:07:38.982 INFO:journalctl@ceph.mds.i.smithi131.stdout:Jun 23 01:07:38 smithi131 conmon[120686]: starting mds.i at 2023-06-23T01:07:38.982 INFO:journalctl@ceph.mds.i.smithi131.stdout:Jun 23 01:07:38 smithi131 bash[120647]: 664b33e1d9aae8dafc9bfd9b20d02393fb0e6021ddd420bb02f1fb849fa8213a 2023-06-23T01:07:38.983 INFO:journalctl@ceph.mds.i.smithi131.stdout:Jun 23 01:07:38 smithi131 systemd[1]: Started Ceph mds.i for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:39.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:38 smithi046 ceph-mon[84949]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:39.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:38 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:39.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:38 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:39.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:38 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:39.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:38 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:39.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:38 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:39.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:38 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:39.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:38 smithi092 ceph-mon[85670]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:39.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:39.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:39.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:39.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:39.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:39.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:38 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:39.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:39.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:38 smithi131 ceph-mon[86495]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:39.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:38 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:39.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:38 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:39.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:38 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:39.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:38 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:39.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:38 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:39.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:38 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.545+0000 7f9729249700 1 -- 172.21.15.46:0/561184048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97240fbbc0 msgr2=0x7f97240fbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.545+0000 7f9729249700 1 --2- 172.21.15.46:0/561184048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97240fbbc0 0x7f97240fbfd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f97140049c0 tx=0x7f9714009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.545+0000 7f9729249700 1 -- 172.21.15.46:0/561184048 shutdown_connections 2023-06-23T01:07:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.545+0000 7f9729249700 1 --2- 172.21.15.46:0/561184048 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97240fd820 0x7f97241040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.545+0000 7f9729249700 1 --2- 172.21.15.46:0/561184048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97240fbbc0 0x7f97240fbfd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.545+0000 7f9729249700 1 --2- 172.21.15.46:0/561184048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97240fce70 0x7f97240fd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.545+0000 7f9729249700 1 -- 172.21.15.46:0/561184048 >> 172.21.15.46:0/561184048 conn(0x7f97240f7300 msgr2=0x7f97240f9750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.545+0000 7f9729249700 1 -- 172.21.15.46:0/561184048 shutdown_connections 2023-06-23T01:07:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.545+0000 7f9729249700 1 -- 172.21.15.46:0/561184048 wait complete. 2023-06-23T01:07:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.545+0000 7f9729249700 1 Processor -- start 2023-06-23T01:07:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.545+0000 7f9729249700 1 -- start start 2023-06-23T01:07:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f9729249700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97240fbbc0 0x7f9724066c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f9729249700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97240fce70 0x7f9724112650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f9729249700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97240fd820 0x7f9724114c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f9729249700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9724115430 con 0x7f97240fd820 2023-06-23T01:07:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f9729249700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97241155a0 con 0x7f97240fce70 2023-06-23T01:07:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f9729249700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9724115760 con 0x7f97240fbbc0 2023-06-23T01:07:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f972359e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97240fd820 0x7f9724114c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f972359e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97240fd820 0x7f9724114c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54422/0 (socket says 172.21.15.46:54422) 2023-06-23T01:07:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f972359e700 1 -- 172.21.15.46:0/858931046 learned_addr learned my addr 172.21.15.46:0/858931046 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f972259c700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97240fce70 0x7f9724112650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f9722d9d700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97240fbbc0 0x7f9724066c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f972359e700 1 -- 172.21.15.46:0/858931046 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97240fbbc0 msgr2=0x7f9724066c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f972359e700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97240fbbc0 0x7f9724066c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f972359e700 1 -- 172.21.15.46:0/858931046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97240fce70 msgr2=0x7f9724112650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f972359e700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97240fce70 0x7f9724112650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f972359e700 1 -- 172.21.15.46:0/858931046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714005040 con 0x7f97240fd820 2023-06-23T01:07:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f972359e700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97240fd820 0x7f9724114c90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f971c001d00 tx=0x7f971c009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f970b7fe700 1 -- 172.21.15.46:0/858931046 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c001e60 con 0x7f97240fd820 2023-06-23T01:07:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f970b7fe700 1 -- 172.21.15.46:0/858931046 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f971c00f040 con 0x7f97240fd820 2023-06-23T01:07:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f9729249700 1 -- 172.21.15.46:0/858931046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9724115a40 con 0x7f97240fd820 2023-06-23T01:07:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.546+0000 7f970b7fe700 1 -- 172.21.15.46:0/858931046 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c00de50 con 0x7f97240fd820 2023-06-23T01:07:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.547+0000 7f9729249700 1 -- 172.21.15.46:0/858931046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9724115fb0 con 0x7f97240fd820 2023-06-23T01:07:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.547+0000 7f9729249700 1 -- 172.21.15.46:0/858931046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9724102bc0 con 0x7f97240fd820 2023-06-23T01:07:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.548+0000 7f970b7fe700 1 -- 172.21.15.46:0/858931046 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f971c007b30 con 0x7f97240fd820 2023-06-23T01:07:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.549+0000 7f970b7fe700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f970c09b700 0x7f970c09dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.549+0000 7f9722d9d700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f970c09b700 0x7f970c09dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.549+0000 7f970b7fe700 1 -- 172.21.15.46:0/858931046 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f971c0c4680 con 0x7f97240fd820 2023-06-23T01:07:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.549+0000 7f9722d9d700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f970c09b700 0x7f970c09dbb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9714009b80 tx=0x7f9714009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.555+0000 7f970b7fe700 1 -- 172.21.15.46:0/858931046 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f971c090410 con 0x7f97240fd820 2023-06-23T01:07:39.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.807+0000 7f9729249700 1 -- 172.21.15.46:0/858931046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9724107280 con 0x7f97240fd820 2023-06-23T01:07:39.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.807+0000 7f970b7fe700 1 -- 172.21.15.46:0/858931046 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7321 (secure 0 0 0) 0x7f971c093a30 con 0x7f97240fd820 2023-06-23T01:07:39.810 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:39.810 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2158355110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2158355110},{"type":"v1","addr":"172.21.15.46:6835","nonce":2158355110}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2977827695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2977827695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2977827695}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/3255240190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":3255240190},{"type":"v1","addr":"172.21.15.131:6833","nonce":3255240190}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1677883305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1677883305},{"type":"v1","addr":"172.21.15.92:6833","nonce":1677883305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/653916810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":653916810},{"type":"v1","addr":"172.21.15.131:6835","nonce":653916810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/2078557791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2078557791},{"type":"v1","addr":"172.21.15.92:6835","nonce":2078557791}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2849145797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2849145797},{"type":"v1","addr":"172.21.15.131:6837","nonce":2849145797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6839/3287494359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":3287494359},{"type":"v1","addr":"172.21.15.131:6839","nonce":3287494359}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/447743959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":447743959},{"type":"v1","addr":"172.21.15.92:6837","nonce":447743959}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-06-23T01:07:39.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 -- 172.21.15.46:0/858931046 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f970c09b700 msgr2=0x7f970c09dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:39.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f970c09b700 0x7f970c09dbb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9714009b80 tx=0x7f9714009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:39.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 -- 172.21.15.46:0/858931046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97240fd820 msgr2=0x7f9724114c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:39.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97240fd820 0x7f9724114c90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f971c001d00 tx=0x7f971c009f30 comp rx=0 tx=0).stop 2023-06-23T01:07:39.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 -- 172.21.15.46:0/858931046 shutdown_connections 2023-06-23T01:07:39.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97240fd820 0x7f9724114c90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:39.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f97240fbbc0 0x7f9724066c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:39.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f970c09b700 0x7f970c09dbb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:39.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 --2- 172.21.15.46:0/858931046 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f97240fce70 0x7f9724112650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:39.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 -- 172.21.15.46:0/858931046 >> 172.21.15.46:0/858931046 conn(0x7f97240f7300 msgr2=0x7f97240f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:39.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 -- 172.21.15.46:0/858931046 shutdown_connections 2023-06-23T01:07:39.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:39.810+0000 7f9729249700 1 -- 172.21.15.46:0/858931046 wait complete. 2023-06-23T01:07:39.814 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 10 2023-06-23T01:07:40.083 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:39 smithi046 ceph-mon[84949]: Deploying daemon mds.g on smithi046 2023-06-23T01:07:40.083 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:39 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.131:6838/3287494359,v1:172.21.15.131:6839/3287494359] up:boot 2023-06-23T01:07:40.083 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:39 smithi046 ceph-mon[84949]: fsmap 9 up:standby 2023-06-23T01:07:40.084 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:39 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-06-23T01:07:40.084 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:39 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/858931046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:40.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:39 smithi092 ceph-mon[85670]: Deploying daemon mds.g on smithi046 2023-06-23T01:07:40.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:39 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.131:6838/3287494359,v1:172.21.15.131:6839/3287494359] up:boot 2023-06-23T01:07:40.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:39 smithi092 ceph-mon[85670]: fsmap 9 up:standby 2023-06-23T01:07:40.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:39 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-06-23T01:07:40.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:39 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/858931046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:40.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:39 smithi131 ceph-mon[86495]: Deploying daemon mds.g on smithi046 2023-06-23T01:07:40.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:39 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.131:6838/3287494359,v1:172.21.15.131:6839/3287494359] up:boot 2023-06-23T01:07:40.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:39 smithi131 ceph-mon[86495]: fsmap 9 up:standby 2023-06-23T01:07:40.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:39 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-06-23T01:07:40.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:39 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/858931046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:41.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:40 smithi046 ceph-mon[84949]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:41.318 INFO:journalctl@ceph.mds.g.smithi046.stdout:Jun 23 01:07:41 smithi046 conmon[129976]: starting mds.g at 2023-06-23T01:07:41.318 INFO:journalctl@ceph.mds.g.smithi046.stdout:Jun 23 01:07:41 smithi046 systemd[1]: Started Ceph mds.g for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:41.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:40 smithi092 ceph-mon[85670]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:41.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:40 smithi131 ceph-mon[86495]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:41.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.959+0000 7f0c57d6e700 1 -- 172.21.15.46:0/639564370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c50066880 msgr2=0x7f0c500ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.959+0000 7f0c57d6e700 1 --2- 172.21.15.46:0/639564370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c50066880 0x7f0c500ff830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0049c0 tx=0x7f0c4c009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.959+0000 7f0c57d6e700 1 -- 172.21.15.46:0/639564370 shutdown_connections 2023-06-23T01:07:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.959+0000 7f0c57d6e700 1 --2- 172.21.15.46:0/639564370 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c501028f0 0x7f0c50104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.959+0000 7f0c57d6e700 1 --2- 172.21.15.46:0/639564370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c50066880 0x7f0c500ff830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.959+0000 7f0c57d6e700 1 --2- 172.21.15.46:0/639564370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c500ffe00 0x7f0c50102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.959+0000 7f0c57d6e700 1 -- 172.21.15.46:0/639564370 >> 172.21.15.46:0/639564370 conn(0x7f0c500f72a0 msgr2=0x7f0c500f9710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.959+0000 7f0c57d6e700 1 -- 172.21.15.46:0/639564370 shutdown_connections 2023-06-23T01:07:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.959+0000 7f0c57d6e700 1 -- 172.21.15.46:0/639564370 wait complete. 2023-06-23T01:07:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.959+0000 7f0c57d6e700 1 Processor -- start 2023-06-23T01:07:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.959+0000 7f0c57d6e700 1 -- start start 2023-06-23T01:07:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c57d6e700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c50066880 0x7f0c5010d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c57d6e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c500ffe00 0x7f0c5010dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c57d6e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c501028f0 0x7f0c50114de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c57d6e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c501153f0 con 0x7f0c501028f0 2023-06-23T01:07:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c57d6e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c50115560 con 0x7f0c500ffe00 2023-06-23T01:07:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c57d6e700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c50115720 con 0x7f0c50066880 2023-06-23T01:07:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c5630b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c501028f0 0x7f0c50114de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c5630b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c501028f0 0x7f0c50114de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54438/0 (socket says 172.21.15.46:54438) 2023-06-23T01:07:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c5630b700 1 -- 172.21.15.46:0/3492195719 learned_addr learned my addr 172.21.15.46:0/3492195719 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c55b0a700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c50066880 0x7f0c5010d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c55309700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c500ffe00 0x7f0c5010dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c5630b700 1 -- 172.21.15.46:0/3492195719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c50066880 msgr2=0x7f0c5010d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c5630b700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c50066880 0x7f0c5010d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c5630b700 1 -- 172.21.15.46:0/3492195719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c500ffe00 msgr2=0x7f0c5010dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c5630b700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c500ffe00 0x7f0c5010dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c5630b700 1 -- 172.21.15.46:0/3492195719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c005040 con 0x7f0c501028f0 2023-06-23T01:07:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c55309700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c500ffe00 0x7f0c5010dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:07:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c5630b700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c501028f0 0x7f0c50114de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0c48001d00 tx=0x7f0c48009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c467fc700 1 -- 172.21.15.46:0/3492195719 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48001e60 con 0x7f0c501028f0 2023-06-23T01:07:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c467fc700 1 -- 172.21.15.46:0/3492195719 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0c4800f040 con 0x7f0c501028f0 2023-06-23T01:07:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c57d6e700 1 -- 172.21.15.46:0/3492195719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c50115a00 con 0x7f0c501028f0 2023-06-23T01:07:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.960+0000 7f0c467fc700 1 -- 172.21.15.46:0/3492195719 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c480090d0 con 0x7f0c501028f0 2023-06-23T01:07:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.961+0000 7f0c57d6e700 1 -- 172.21.15.46:0/3492195719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c50115f20 con 0x7f0c501028f0 2023-06-23T01:07:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.961+0000 7f0c57d6e700 1 -- 172.21.15.46:0/3492195719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c500f8dd0 con 0x7f0c501028f0 2023-06-23T01:07:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.963+0000 7f0c467fc700 1 -- 172.21.15.46:0/3492195719 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f0c48009500 con 0x7f0c501028f0 2023-06-23T01:07:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.963+0000 7f0c467fc700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f0c3c09b670 0x7f0c3c09db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.963+0000 7f0c55b0a700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f0c3c09b670 0x7f0c3c09db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.963+0000 7f0c467fc700 1 -- 172.21.15.46:0/3492195719 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f0c480c4d70 con 0x7f0c501028f0 2023-06-23T01:07:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.964+0000 7f0c55b0a700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f0c3c09b670 0x7f0c3c09db20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c004fb0 tx=0x7f0c4c009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:41.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:41.966+0000 7f0c467fc700 1 -- 172.21.15.46:0/3492195719 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f0c48090b00 con 0x7f0c501028f0 2023-06-23T01:07:42.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.218+0000 7f0c57d6e700 1 -- 172.21.15.46:0/3492195719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c501082f0 con 0x7f0c501028f0 2023-06-23T01:07:42.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.219+0000 7f0c467fc700 1 -- 172.21.15.46:0/3492195719 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8073 (secure 0 0 0) 0x7f0c48094120 con 0x7f0c501028f0 2023-06-23T01:07:42.221 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:42.222 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2158355110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2158355110},{"type":"v1","addr":"172.21.15.46:6835","nonce":2158355110}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2977827695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2977827695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2977827695}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/50961189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":50961189},{"type":"v1","addr":"172.21.15.46:6839","nonce":50961189}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/3255240190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":3255240190},{"type":"v1","addr":"172.21.15.131:6833","nonce":3255240190}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1677883305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1677883305},{"type":"v1","addr":"172.21.15.92:6833","nonce":1677883305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/653916810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":653916810},{"type":"v1","addr":"172.21.15.131:6835","nonce":653916810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/2078557791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2078557791},{"type":"v1","addr":"172.21.15.92:6835","nonce":2078557791}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2849145797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2849145797},{"type":"v1","addr":"172.21.15.131:6837","nonce":2849145797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6839/3287494359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":3287494359},{"type":"v1","addr":"172.21.15.131:6839","nonce":3287494359}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/447743959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":447743959},{"type":"v1","addr":"172.21.15.92:6837","nonce":447743959}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-06-23T01:07:42.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 -- 172.21.15.46:0/3492195719 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f0c3c09b670 msgr2=0x7f0c3c09db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f0c3c09b670 0x7f0c3c09db20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c004fb0 tx=0x7f0c4c009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 -- 172.21.15.46:0/3492195719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c501028f0 msgr2=0x7f0c50114de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c501028f0 0x7f0c50114de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0c48001d00 tx=0x7f0c48009f30 comp rx=0 tx=0).stop 2023-06-23T01:07:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 -- 172.21.15.46:0/3492195719 shutdown_connections 2023-06-23T01:07:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c501028f0 0x7f0c50114de0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:42.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f0c50066880 0x7f0c5010d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:42.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f0c3c09b670 0x7f0c3c09db20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:42.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 --2- 172.21.15.46:0/3492195719 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f0c500ffe00 0x7f0c5010dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:42.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 -- 172.21.15.46:0/3492195719 >> 172.21.15.46:0/3492195719 conn(0x7f0c500f72a0 msgr2=0x7f0c500fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:42.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 -- 172.21.15.46:0/3492195719 shutdown_connections 2023-06-23T01:07:42.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:42.221+0000 7f0c57d6e700 1 -- 172.21.15.46:0/3492195719 wait complete. 2023-06-23T01:07:42.225 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 11 2023-06-23T01:07:42.372 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:42.373 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:42.373 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:42.373 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:42.373 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:42.374 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:42.374 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:42 smithi046 ceph-mon[84949]: Deploying daemon mds.a on smithi046 2023-06-23T01:07:42.374 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:42 smithi046 ceph-mon[84949]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:42.374 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:42 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.46:6838/50961189,v1:172.21.15.46:6839/50961189] up:boot 2023-06-23T01:07:42.374 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:42 smithi046 ceph-mon[84949]: fsmap 10 up:standby 2023-06-23T01:07:42.374 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:42 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-06-23T01:07:42.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:42.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:42 smithi092 ceph-mon[85670]: Deploying daemon mds.a on smithi046 2023-06-23T01:07:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:42 smithi092 ceph-mon[85670]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:42 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.46:6838/50961189,v1:172.21.15.46:6839/50961189] up:boot 2023-06-23T01:07:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:42 smithi092 ceph-mon[85670]: fsmap 10 up:standby 2023-06-23T01:07:42.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:42 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-06-23T01:07:42.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:42.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:42.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:42.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:42.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:42.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:42.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:42 smithi131 ceph-mon[86495]: Deploying daemon mds.a on smithi046 2023-06-23T01:07:42.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:42 smithi131 ceph-mon[86495]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:42.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:42 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.46:6838/50961189,v1:172.21.15.46:6839/50961189] up:boot 2023-06-23T01:07:42.483 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:42 smithi131 ceph-mon[86495]: fsmap 10 up:standby 2023-06-23T01:07:42.484 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:42 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-06-23T01:07:43.068 INFO:journalctl@ceph.mds.a.smithi046.stdout:Jun 23 01:07:42 smithi046 systemd[1]: Starting Ceph mds.a for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:43.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:43 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/3492195719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:43.482 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:43 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/3492195719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:43.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:43 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3492195719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:43.568 INFO:journalctl@ceph.mds.a.smithi046.stdout:Jun 23 01:07:43 smithi046 ceph-mds[130600]: -- 172.21.15.46:0/3321349131 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556374d1c9c0 con 0x556374cf2c00 2023-06-23T01:07:43.569 INFO:journalctl@ceph.mds.a.smithi046.stdout:Jun 23 01:07:43 smithi046 bash[130536]: c13f62d33d050f80439dc37000d78379628f9991dd80f86a2c588deec8bd3a2b 2023-06-23T01:07:43.569 INFO:journalctl@ceph.mds.a.smithi046.stdout:Jun 23 01:07:43 smithi046 conmon[130576]: starting mds.a at 2023-06-23T01:07:43.569 INFO:journalctl@ceph.mds.a.smithi046.stdout:Jun 23 01:07:43 smithi046 systemd[1]: Started Ceph mds.a for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:44.236 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:44.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.376+0000 7f56958cc700 1 -- 172.21.15.46:0/1449686240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56900fd600 msgr2=0x7f56900fdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:44.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.376+0000 7f56958cc700 1 --2- 172.21.15.46:0/1449686240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56900fd600 0x7f56900fdff0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f56840049c0 tx=0x7f5684009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:44.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.376+0000 7f56958cc700 1 -- 172.21.15.46:0/1449686240 shutdown_connections 2023-06-23T01:07:44.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.376+0000 7f56958cc700 1 --2- 172.21.15.46:0/1449686240 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56900fccb0 0x7f56900fd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:44.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.376+0000 7f56958cc700 1 --2- 172.21.15.46:0/1449686240 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56900fe720 0x7f5690105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:44.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.376+0000 7f56958cc700 1 --2- 172.21.15.46:0/1449686240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56900fd600 0x7f56900fdff0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:44.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.376+0000 7f56958cc700 1 -- 172.21.15.46:0/1449686240 >> 172.21.15.46:0/1449686240 conn(0x7f56900f84a0 msgr2=0x7f56900fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:44.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.377+0000 7f56958cc700 1 -- 172.21.15.46:0/1449686240 shutdown_connections 2023-06-23T01:07:44.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.377+0000 7f56958cc700 1 -- 172.21.15.46:0/1449686240 wait complete. 2023-06-23T01:07:44.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.377+0000 7f56958cc700 1 Processor -- start 2023-06-23T01:07:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.377+0000 7f56958cc700 1 -- start start 2023-06-23T01:07:44.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.377+0000 7f56958cc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56900fccb0 0x7f569010d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:44.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.377+0000 7f56958cc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56900fd600 0x7f569010dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:44.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.377+0000 7f56958cc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56900fe720 0x7f5690114da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:44.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.377+0000 7f56958cc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56901153b0 con 0x7f56900fccb0 2023-06-23T01:07:44.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.377+0000 7f56958cc700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5690115520 con 0x7f56900fe720 2023-06-23T01:07:44.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.377+0000 7f56958cc700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56901156e0 con 0x7f56900fd600 2023-06-23T01:07:44.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56900fccb0 0x7f569010d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:44.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56900fccb0 0x7f569010d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60382/0 (socket says 172.21.15.46:60382) 2023-06-23T01:07:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568f7fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56900fe720 0x7f5690114da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568effd700 1 -- 172.21.15.46:0/4033901488 learned_addr learned my addr 172.21.15.46:0/4033901488 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568e7fc700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56900fd600 0x7f569010dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568effd700 1 -- 172.21.15.46:0/4033901488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56900fd600 msgr2=0x7f569010dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568effd700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56900fd600 0x7f569010dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568effd700 1 -- 172.21.15.46:0/4033901488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56900fe720 msgr2=0x7f5690114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568effd700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56900fe720 0x7f5690114da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568effd700 1 -- 172.21.15.46:0/4033901488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5684005040 con 0x7f56900fccb0 2023-06-23T01:07:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568f7fe700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56900fe720 0x7f5690114da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:07:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f568effd700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56900fccb0 0x7f569010d7d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5680002ad0 tx=0x7f56800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f56777fe700 1 -- 172.21.15.46:0/4033901488 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5680007cc0 con 0x7f56900fccb0 2023-06-23T01:07:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f56958cc700 1 -- 172.21.15.46:0/4033901488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56901159c0 con 0x7f56900fccb0 2023-06-23T01:07:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f56777fe700 1 -- 172.21.15.46:0/4033901488 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f568000f040 con 0x7f56900fccb0 2023-06-23T01:07:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f56777fe700 1 -- 172.21.15.46:0/4033901488 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568000d880 con 0x7f56900fccb0 2023-06-23T01:07:44.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.378+0000 7f56958cc700 1 -- 172.21.15.46:0/4033901488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5690115f80 con 0x7f56900fccb0 2023-06-23T01:07:44.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.379+0000 7f56958cc700 1 -- 172.21.15.46:0/4033901488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5690103cf0 con 0x7f56900fccb0 2023-06-23T01:07:44.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.380+0000 7f56777fe700 1 -- 172.21.15.46:0/4033901488 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f568000dcb0 con 0x7f56900fccb0 2023-06-23T01:07:44.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.381+0000 7f56777fe700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f567809b680 0x7f567809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:44.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.381+0000 7f568e7fc700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f567809b680 0x7f567809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:44.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.381+0000 7f56777fe700 1 -- 172.21.15.46:0/4033901488 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f56800c4de0 con 0x7f56900fccb0 2023-06-23T01:07:44.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.381+0000 7f568e7fc700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f567809b680 0x7f567809db30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5684009b30 tx=0x7f5684009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:44.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.384+0000 7f56777fe700 1 -- 172.21.15.46:0/4033901488 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f5680090a40 con 0x7f56900fccb0 2023-06-23T01:07:44.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:44.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:44.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:44.568 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:44.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:44.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:44.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:44 smithi046 ceph-mon[84949]: Deploying daemon mds.b on smithi092 2023-06-23T01:07:44.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:44 smithi046 ceph-mon[84949]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:44.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:44 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.46:6840/4117518130,v1:172.21.15.46:6841/4117518130] up:boot 2023-06-23T01:07:44.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:44 smithi046 ceph-mon[84949]: fsmap 11 up:standby 2023-06-23T01:07:44.569 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:44 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-06-23T01:07:44.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.630+0000 7f56958cc700 1 -- 172.21.15.46:0/4033901488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5690108320 con 0x7f56900fccb0 2023-06-23T01:07:44.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.631+0000 7f56777fe700 1 -- 172.21.15.46:0/4033901488 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8831 (secure 0 0 0) 0x7f5680094060 con 0x7f56900fccb0 2023-06-23T01:07:44.633 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:44.634 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2158355110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2158355110},{"type":"v1","addr":"172.21.15.46:6835","nonce":2158355110}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2977827695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2977827695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2977827695}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/50961189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":50961189},{"type":"v1","addr":"172.21.15.46:6839","nonce":50961189}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/4117518130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":4117518130},{"type":"v1","addr":"172.21.15.46:6841","nonce":4117518130}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/3255240190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":3255240190},{"type":"v1","addr":"172.21.15.131:6833","nonce":3255240190}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1677883305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1677883305},{"type":"v1","addr":"172.21.15.92:6833","nonce":1677883305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/653916810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":653916810},{"type":"v1","addr":"172.21.15.131:6835","nonce":653916810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/2078557791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2078557791},{"type":"v1","addr":"172.21.15.92:6835","nonce":2078557791}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2849145797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2849145797},{"type":"v1","addr":"172.21.15.131:6837","nonce":2849145797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6839/3287494359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":3287494359},{"type":"v1","addr":"172.21.15.131:6839","nonce":3287494359}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/447743959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":447743959},{"type":"v1","addr":"172.21.15.92:6837","nonce":447743959}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-06-23T01:07:44.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 -- 172.21.15.46:0/4033901488 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f567809b680 msgr2=0x7f567809db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:44.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f567809b680 0x7f567809db30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5684009b30 tx=0x7f5684009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:44.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 -- 172.21.15.46:0/4033901488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56900fccb0 msgr2=0x7f569010d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:44.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56900fccb0 0x7f569010d7d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5680002ad0 tx=0x7f56800046b0 comp rx=0 tx=0).stop 2023-06-23T01:07:44.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 -- 172.21.15.46:0/4033901488 shutdown_connections 2023-06-23T01:07:44.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f567809b680 0x7f567809db30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:44.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f56900fe720 0x7f5690114da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:44.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56900fccb0 0x7f569010d7d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 --2- 172.21.15.46:0/4033901488 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f56900fd600 0x7f569010dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 -- 172.21.15.46:0/4033901488 >> 172.21.15.46:0/4033901488 conn(0x7f56900f84a0 msgr2=0x7f56900fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 -- 172.21.15.46:0/4033901488 shutdown_connections 2023-06-23T01:07:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:44.633+0000 7f56958cc700 1 -- 172.21.15.46:0/4033901488 wait complete. 2023-06-23T01:07:44.638 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2023-06-23T01:07:44.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:44.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:44.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:44.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:44.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:44.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:44.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:44 smithi092 ceph-mon[85670]: Deploying daemon mds.b on smithi092 2023-06-23T01:07:44.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:44 smithi092 ceph-mon[85670]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:44.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:44 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.46:6840/4117518130,v1:172.21.15.46:6841/4117518130] up:boot 2023-06-23T01:07:44.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:44 smithi092 ceph-mon[85670]: fsmap 11 up:standby 2023-06-23T01:07:44.641 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:44 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-06-23T01:07:44.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:44.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:44.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:44.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-06-23T01:07:44.732 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-06-23T01:07:44.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:44.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:44 smithi131 ceph-mon[86495]: Deploying daemon mds.b on smithi092 2023-06-23T01:07:44.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:44 smithi131 ceph-mon[86495]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:44.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:44 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.46:6840/4117518130,v1:172.21.15.46:6841/4117518130] up:boot 2023-06-23T01:07:44.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:44 smithi131 ceph-mon[86495]: fsmap 11 up:standby 2023-06-23T01:07:44.733 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:44 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-06-23T01:07:45.330 INFO:journalctl@ceph.mds.b.smithi092.stdout:Jun 23 01:07:44 smithi092 systemd[1]: Starting Ceph mds.b for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:45.639 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:45 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/4033901488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:45.640 INFO:journalctl@ceph.mds.b.smithi092.stdout:Jun 23 01:07:45 smithi092 ceph-mds[118282]: -- 172.21.15.92:0/1415422205 <== mon.2 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e9789d69c0 con 0x55e9789ad000 2023-06-23T01:07:45.640 INFO:journalctl@ceph.mds.b.smithi092.stdout:Jun 23 01:07:45 smithi092 conmon[118258]: starting mds.b at 2023-06-23T01:07:45.640 INFO:journalctl@ceph.mds.b.smithi092.stdout:Jun 23 01:07:45 smithi092 bash[118218]: e83e5feddb34846b5402cbb320ccc5de46abf809423d5d252361666ec72c46d9 2023-06-23T01:07:45.640 INFO:journalctl@ceph.mds.b.smithi092.stdout:Jun 23 01:07:45 smithi092 systemd[1]: Started Ceph mds.b for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:45.731 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:45 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/4033901488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:45.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:45 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/4033901488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:46.648 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:07:46.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 -- 172.21.15.46:0/1655560774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8180fbbc0 msgr2=0x7ff8180fbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 --2- 172.21.15.46:0/1655560774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8180fbbc0 0x7ff8180fbfd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff8080049c0 tx=0x7ff808009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 -- 172.21.15.46:0/1655560774 shutdown_connections 2023-06-23T01:07:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 --2- 172.21.15.46:0/1655560774 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8180fd820 0x7ff8181040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 --2- 172.21.15.46:0/1655560774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8180fbbc0 0x7ff8180fbfd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 --2- 172.21.15.46:0/1655560774 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8180fce70 0x7ff8180fd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 -- 172.21.15.46:0/1655560774 >> 172.21.15.46:0/1655560774 conn(0x7ff8180f7300 msgr2=0x7ff8180f9750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 -- 172.21.15.46:0/1655560774 shutdown_connections 2023-06-23T01:07:46.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 -- 172.21.15.46:0/1655560774 wait complete. 2023-06-23T01:07:46.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 Processor -- start 2023-06-23T01:07:46.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 -- start start 2023-06-23T01:07:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.772+0000 7ff81e68f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8180fbbc0 0x7ff81810d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81e68f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8180fce70 0x7ff81810ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81e68f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8180fd820 0x7ff818114e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81e68f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff818115490 con 0x7ff8180fd820 2023-06-23T01:07:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81e68f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff818115600 con 0x7ff8180fce70 2023-06-23T01:07:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81e68f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8181157c0 con 0x7ff8180fbbc0 2023-06-23T01:07:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81cc2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8180fd820 0x7ff818114e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81cc2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8180fd820 0x7ff818114e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60402/0 (socket says 172.21.15.46:60402) 2023-06-23T01:07:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81cc2c700 1 -- 172.21.15.46:0/3163789047 learned_addr learned my addr 172.21.15.46:0/3163789047 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff8177fe700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8180fce70 0x7ff81810ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff817fff700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8180fbbc0 0x7ff81810d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81cc2c700 1 -- 172.21.15.46:0/3163789047 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8180fbbc0 msgr2=0x7ff81810d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81cc2c700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8180fbbc0 0x7ff81810d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81cc2c700 1 -- 172.21.15.46:0/3163789047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8180fce70 msgr2=0x7ff81810ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81cc2c700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8180fce70 0x7ff81810ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81cc2c700 1 -- 172.21.15.46:0/3163789047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff808005040 con 0x7ff8180fd820 2023-06-23T01:07:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff8177fe700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8180fce70 0x7ff81810ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:07:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81cc2c700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8180fd820 0x7ff818114e80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff810001d00 tx=0x7ff810009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff814ff9700 1 -- 172.21.15.46:0/3163789047 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810001e60 con 0x7ff8180fd820 2023-06-23T01:07:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff814ff9700 1 -- 172.21.15.46:0/3163789047 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff81000f040 con 0x7ff8180fd820 2023-06-23T01:07:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff81e68f700 1 -- 172.21.15.46:0/3163789047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff818115aa0 con 0x7ff8180fd820 2023-06-23T01:07:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.773+0000 7ff814ff9700 1 -- 172.21.15.46:0/3163789047 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81000de50 con 0x7ff8180fd820 2023-06-23T01:07:46.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.774+0000 7ff81e68f700 1 -- 172.21.15.46:0/3163789047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff818115fc0 con 0x7ff8180fd820 2023-06-23T01:07:46.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.775+0000 7ff81e68f700 1 -- 172.21.15.46:0/3163789047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff818103df0 con 0x7ff8180fd820 2023-06-23T01:07:46.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.776+0000 7ff814ff9700 1 -- 172.21.15.46:0/3163789047 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7ff810007b30 con 0x7ff8180fd820 2023-06-23T01:07:46.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.777+0000 7ff814ff9700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff80409b680 0x7ff80409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:46.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.777+0000 7ff814ff9700 1 -- 172.21.15.46:0/3163789047 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7ff8100c4720 con 0x7ff8180fd820 2023-06-23T01:07:46.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.777+0000 7ff817fff700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff80409b680 0x7ff80409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:46.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.778+0000 7ff817fff700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff80409b680 0x7ff80409db30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff808009b80 tx=0x7ff808009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:46.782+0000 7ff814ff9700 1 -- 172.21.15.46:0/3163789047 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7ff8100904b0 con 0x7ff8180fd820 2023-06-23T01:07:46.889 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:46 smithi092 ceph-mon[85670]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:46.890 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:46 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.92:6838/3591784213,v1:172.21.15.92:6839/3591784213] up:boot 2023-06-23T01:07:46.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:46 smithi092 ceph-mon[85670]: fsmap 12 up:standby 2023-06-23T01:07:46.891 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:46 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-06-23T01:07:46.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:46.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:46 smithi131 ceph-mon[86495]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:46.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:46.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:46 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.92:6838/3591784213,v1:172.21.15.92:6839/3591784213] up:boot 2023-06-23T01:07:46.983 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:46 smithi131 ceph-mon[86495]: fsmap 12 up:standby 2023-06-23T01:07:46.984 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:46 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-06-23T01:07:47.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.025+0000 7ff81e68f700 1 -- 172.21.15.46:0/3163789047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff818107280 con 0x7ff8180fd820 2023-06-23T01:07:47.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.027+0000 7ff814ff9700 1 -- 172.21.15.46:0/3163789047 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9589 (secure 0 0 0) 0x7ff810093ad0 con 0x7ff8180fd820 2023-06-23T01:07:47.029 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:47.029 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14577,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2158355110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2158355110},{"type":"v1","addr":"172.21.15.46:6835","nonce":2158355110}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/2977827695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":2977827695},{"type":"v1","addr":"172.21.15.46:6837","nonce":2977827695}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/50961189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":50961189},{"type":"v1","addr":"172.21.15.46:6839","nonce":50961189}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/4117518130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":4117518130},{"type":"v1","addr":"172.21.15.46:6841","nonce":4117518130}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6833/3255240190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6832","nonce":3255240190},{"type":"v1","addr":"172.21.15.131:6833","nonce":3255240190}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6833/1677883305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":1677883305},{"type":"v1","addr":"172.21.15.92:6833","nonce":1677883305}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24454,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6835/653916810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6834","nonce":653916810},{"type":"v1","addr":"172.21.15.131:6835","nonce":653916810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6835/2078557791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":2078557791},{"type":"v1","addr":"172.21.15.92:6835","nonce":2078557791}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24466,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6837/2849145797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6836","nonce":2849145797},{"type":"v1","addr":"172.21.15.131:6837","nonce":2849145797}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.131:6839/3287494359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6838","nonce":3287494359},{"type":"v1","addr":"172.21.15.131:6839","nonce":3287494359}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/447743959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6836","nonce":447743959},{"type":"v1","addr":"172.21.15.92:6837","nonce":447743959}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24530,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6839/3591784213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6838","nonce":3591784213},{"type":"v1","addr":"172.21.15.92:6839","nonce":3591784213}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-06-23T01:07:47.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 -- 172.21.15.46:0/3163789047 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff80409b680 msgr2=0x7ff80409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff80409b680 0x7ff80409db30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff808009b80 tx=0x7ff808009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:47.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 -- 172.21.15.46:0/3163789047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8180fd820 msgr2=0x7ff818114e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8180fd820 0x7ff818114e80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff810001d00 tx=0x7ff810009f30 comp rx=0 tx=0).stop 2023-06-23T01:07:47.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 -- 172.21.15.46:0/3163789047 shutdown_connections 2023-06-23T01:07:47.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8180fd820 0x7ff818114e80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7ff8180fbbc0 0x7ff81810d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7ff80409b680 0x7ff80409db30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 --2- 172.21.15.46:0/3163789047 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff8180fce70 0x7ff81810ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 -- 172.21.15.46:0/3163789047 >> 172.21.15.46:0/3163789047 conn(0x7ff8180f7300 msgr2=0x7ff8180f9630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:47.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 -- 172.21.15.46:0/3163789047 shutdown_connections 2023-06-23T01:07:47.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.029+0000 7ff81e68f700 1 -- 172.21.15.46:0/3163789047 wait complete. 2023-06-23T01:07:47.036 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 13 2023-06-23T01:07:47.042 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:47.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:47.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:47.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:47.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:47.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:47.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:46 smithi046 ceph-mon[84949]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:47.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:47.044 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:46 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.92:6838/3591784213,v1:172.21.15.92:6839/3591784213] up:boot 2023-06-23T01:07:47.044 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:46 smithi046 ceph-mon[84949]: fsmap 12 up:standby 2023-06-23T01:07:47.044 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:46 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-06-23T01:07:47.045 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:07:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.164+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/1265345675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1aac0fcc90 msgr2=0x7f1aac0fd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.164+0000 7f1ab2b9d700 1 --2- 172.21.15.46:0/1265345675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1aac0fcc90 0x7f1aac0fd100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1aa00049c0 tx=0x7f1aa0009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.164+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/1265345675 shutdown_connections 2023-06-23T01:07:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.164+0000 7f1ab2b9d700 1 --2- 172.21.15.46:0/1265345675 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1aac0fd640 0x7f1aac103ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.164+0000 7f1ab2b9d700 1 --2- 172.21.15.46:0/1265345675 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1aac0fb9e0 0x7f1aac0fbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.164+0000 7f1ab2b9d700 1 --2- 172.21.15.46:0/1265345675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1aac0fcc90 0x7f1aac0fd100 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.164+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/1265345675 >> 172.21.15.46:0/1265345675 conn(0x7f1aac0f7140 msgr2=0x7f1aac0f9570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.164+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/1265345675 shutdown_connections 2023-06-23T01:07:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.164+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/1265345675 wait complete. 2023-06-23T01:07:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.164+0000 7f1ab2b9d700 1 Processor -- start 2023-06-23T01:07:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.164+0000 7f1ab2b9d700 1 -- start start 2023-06-23T01:07:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab2b9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1aac0fb9e0 0x7f1aac0ff970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab2b9d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1aac0fcc90 0x7f1aac0fdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab2b9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1aac0fd640 0x7f1aac0fe510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab2b9d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac0fec80 con 0x7f1aac0fd640 2023-06-23T01:07:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab2b9d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac0fedf0 con 0x7f1aac0fb9e0 2023-06-23T01:07:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab2b9d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac117760 con 0x7f1aac0fcc90 2023-06-23T01:07:47.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab239c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1aac0fd640 0x7f1aac0fe510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab1b9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1aac0fb9e0 0x7f1aac0ff970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab239c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1aac0fd640 0x7f1aac0fe510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60414/0 (socket says 172.21.15.46:60414) 2023-06-23T01:07:47.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab239c700 1 -- 172.21.15.46:0/2380312289 learned_addr learned my addr 172.21.15.46:0/2380312289 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:47.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab139a700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1aac0fcc90 0x7f1aac0fdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab239c700 1 -- 172.21.15.46:0/2380312289 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1aac0fcc90 msgr2=0x7f1aac0fdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab239c700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1aac0fcc90 0x7f1aac0fdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab239c700 1 -- 172.21.15.46:0/2380312289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1aac0fb9e0 msgr2=0x7f1aac0ff970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab239c700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1aac0fb9e0 0x7f1aac0ff970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab239c700 1 -- 172.21.15.46:0/2380312289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa0005040 con 0x7f1aac0fd640 2023-06-23T01:07:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab239c700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1aac0fd640 0x7f1aac0fe510 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00ec30 tx=0x7f1a9c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1aaa7fc700 1 -- 172.21.15.46:0/2380312289 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c007b30 con 0x7f1aac0fd640 2023-06-23T01:07:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1aaa7fc700 1 -- 172.21.15.46:0/2380312289 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1a9c00f040 con 0x7f1aac0fd640 2023-06-23T01:07:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/2380312289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aac117a10 con 0x7f1aac0fd640 2023-06-23T01:07:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1aaa7fc700 1 -- 172.21.15.46:0/2380312289 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c00c9c0 con 0x7f1aac0fd640 2023-06-23T01:07:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.165+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/2380312289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aac117f30 con 0x7f1aac0fd640 2023-06-23T01:07:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.166+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/2380312289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aac1070f0 con 0x7f1aac0fd640 2023-06-23T01:07:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.168+0000 7f1aaa7fc700 1 -- 172.21.15.46:0/2380312289 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f1a9c009090 con 0x7f1aac0fd640 2023-06-23T01:07:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.169+0000 7f1aaa7fc700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1a9409b680 0x7f1a9409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.169+0000 7f1ab1b9b700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1a9409b680 0x7f1a9409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.169+0000 7f1aaa7fc700 1 -- 172.21.15.46:0/2380312289 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f1a9c0c53e0 con 0x7f1aac0fd640 2023-06-23T01:07:47.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.169+0000 7f1ab1b9b700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1a9409b680 0x7f1a9409db30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009b80 tx=0x7f1aa400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:47.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.172+0000 7f1aaa7fc700 1 -- 172.21.15.46:0/2380312289 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f1a9c0910e0 con 0x7f1aac0fd640 2023-06-23T01:07:47.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.372+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/2380312289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1aac118180 con 0x7f1aac0fd640 2023-06-23T01:07:47.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.373+0000 7f1aaa7fc700 1 -- 172.21.15.46:0/2380312289 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18097 (secure 0 0 0) 0x7f1a9c094700 con 0x7f1aac0fd640 2023-06-23T01:07:47.376 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:47.376 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":77,"fsid":"458642ee-1161-11ee-9b2b-001a4aab830c","created":"2023-06-23T01:00:57.877538+0000","modified":"2023-06-23T01:07:09.444468+0000","last_up_change":"2023-06-23T01:07:07.914869+0000","last_in_change":"2023-06-23T01:06:51.140023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:04:00.299796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f28767ce-a780-4e50-b118-37a8c275c0e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6803","nonce":709285010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6805","nonce":709285010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6809","nonce":709285010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6807","nonce":709285010}]},"public_addr":"172.21.15.46:6803/709285010","cluster_addr":"172.21.15.46:6805/709285010","heartbeat_back_addr":"172.21.15.46:6809/709285010","heartbeat_front_addr":"172.21.15.46:6807/709285010","state":["exists","up"]},{"osd":1,"uuid":"4d4c4c36-d4a1-4b96-9059-adffee04401a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6801","nonce":4266072183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6803","nonce":4266072183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6807","nonce":4266072183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6805","nonce":4266072183}]},"public_addr":"172.21.15.92:6801/4266072183","cluster_addr":"172.21.15.92:6803/4266072183","heartbeat_back_addr":"172.21.15.92:6807/4266072183","heartbeat_front_addr":"172.21.15.92:6805/4266072183","state":["exists","up"]},{"osd":2,"uuid":"ec7b0c60-88ea-4c2b-a34c-96d188e02014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6801","nonce":1787957441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6803","nonce":1787957441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6807","nonce":1787957441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6805","nonce":1787957441}]},"public_addr":"172.21.15.131:6801/1787957441","cluster_addr":"172.21.15.131:6803/1787957441","heartbeat_back_addr":"172.21.15.131:6807/1787957441","heartbeat_front_addr":"172.21.15.131:6805/1787957441","state":["exists","up"]},{"osd":3,"uuid":"32670a23-3286-4538-9628-38d763429cbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6811","nonce":338690444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6813","nonce":338690444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6817","nonce":338690444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6815","nonce":338690444}]},"public_addr":"172.21.15.46:6811/338690444","cluster_addr":"172.21.15.46:6813/338690444","heartbeat_back_addr":"172.21.15.46:6817/338690444","heartbeat_front_addr":"172.21.15.46:6815/338690444","state":["exists","up"]},{"osd":4,"uuid":"0d8f982c-eb74-48a2-949e-d0ccd7732937","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6809","nonce":2024629902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6811","nonce":2024629902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6815","nonce":2024629902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6813","nonce":2024629902}]},"public_addr":"172.21.15.92:6809/2024629902","cluster_addr":"172.21.15.92:6811/2024629902","heartbeat_back_addr":"172.21.15.92:6815/2024629902","heartbeat_front_addr":"172.21.15.92:6813/2024629902","state":["exists","up"]},{"osd":5,"uuid":"52a7160c-4735-4a77-bc08-4d411c282b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6809","nonce":2577961971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6811","nonce":2577961971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6815","nonce":2577961971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6813","nonce":2577961971}]},"public_addr":"172.21.15.131:6809/2577961971","cluster_addr":"172.21.15.131:6811/2577961971","heartbeat_back_addr":"172.21.15.131:6815/2577961971","heartbeat_front_addr":"172.21.15.131:6813/2577961971","state":["exists","up"]},{"osd":6,"uuid":"3f3d2690-f655-4dda-a36c-bc97f9d08a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6819","nonce":558288790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6821","nonce":558288790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6825","nonce":558288790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6823","nonce":558288790}]},"public_addr":"172.21.15.46:6819/558288790","cluster_addr":"172.21.15.46:6821/558288790","heartbeat_back_addr":"172.21.15.46:6825/558288790","heartbeat_front_addr":"172.21.15.46:6823/558288790","state":["exists","up"]},{"osd":7,"uuid":"2eb43609-8921-4891-b0c4-af932bfb69bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6817","nonce":3796540277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6819","nonce":3796540277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6823","nonce":3796540277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6821","nonce":3796540277}]},"public_addr":"172.21.15.92:6817/3796540277","cluster_addr":"172.21.15.92:6819/3796540277","heartbeat_back_addr":"172.21.15.92:6823/3796540277","heartbeat_front_addr":"172.21.15.92:6821/3796540277","state":["exists","up"]},{"osd":8,"uuid":"0139b39e-f933-4685-b056-a6c27fa87fd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6817","nonce":1004740496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6819","nonce":1004740496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6823","nonce":1004740496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6821","nonce":1004740496}]},"public_addr":"172.21.15.131:6817/1004740496","cluster_addr":"172.21.15.131:6819/1004740496","heartbeat_back_addr":"172.21.15.131:6823/1004740496","heartbeat_front_addr":"172.21.15.131:6821/1004740496","state":["exists","up"]},{"osd":9,"uuid":"8bd083b4-f766-4e1b-b959-19d2fa82d77f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6827","nonce":3042170676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6829","nonce":3042170676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6833","nonce":3042170676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6831","nonce":3042170676}]},"public_addr":"172.21.15.46:6827/3042170676","cluster_addr":"172.21.15.46:6829/3042170676","heartbeat_back_addr":"172.21.15.46:6833/3042170676","heartbeat_front_addr":"172.21.15.46:6831/3042170676","state":["exists","up"]},{"osd":10,"uuid":"0b4f58f5-d971-4634-a1b6-896501d3d218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6825","nonce":2375910823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6827","nonce":2375910823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6831","nonce":2375910823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6829","nonce":2375910823}]},"public_addr":"172.21.15.92:6825/2375910823","cluster_addr":"172.21.15.92:6827/2375910823","heartbeat_back_addr":"172.21.15.92:6831/2375910823","heartbeat_front_addr":"172.21.15.92:6829/2375910823","state":["exists","up"]},{"osd":11,"uuid":"7d7a4115-c6e9-4cef-865b-05f57fbcface","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6825","nonce":1716414800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6827","nonce":1716414800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6831","nonce":1716414800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6829","nonce":1716414800}]},"public_addr":"172.21.15.131:6825/1716414800","cluster_addr":"172.21.15.131:6827/1716414800","heartbeat_back_addr":"172.21.15.131:6831/1716414800","heartbeat_front_addr":"172.21.15.131:6829/1716414800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:15.785940+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:36.479807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:57.678238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:04:20.396101+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:04:41.179881+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:01.570524+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:22.313074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:43.129445+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:03.897470+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:25.196569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:46.208577+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:07:06.788656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/685567953":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6801/2331157358":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6800/2331157358":"2023-06-24T01:01:55.786463+0000","172.21.15.46:0/1655911006":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/187730254":"2023-06-24T01:01:32.095580+0000","172.21.15.46:6801/3291477171":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/3127678959":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6800/3291477171":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/411746526":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/360533287":"2023-06-24T01:01:55.786463+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:07:47.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/2380312289 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1a9409b680 msgr2=0x7f1a9409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1a9409b680 0x7f1a9409db30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009b80 tx=0x7f1aa400b040 comp rx=0 tx=0).stop 2023-06-23T01:07:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/2380312289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1aac0fd640 msgr2=0x7f1aac0fe510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1aac0fd640 0x7f1aac0fe510 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00ec30 tx=0x7f1a9c00b040 comp rx=0 tx=0).stop 2023-06-23T01:07:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/2380312289 shutdown_connections 2023-06-23T01:07:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f1a9409b680 0x7f1a9409db30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1aac0fb9e0 0x7f1aac0ff970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1aac0fd640 0x7f1aac0fe510 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 --2- 172.21.15.46:0/2380312289 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f1aac0fcc90 0x7f1aac0fdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/2380312289 >> 172.21.15.46:0/2380312289 conn(0x7f1aac0f7140 msgr2=0x7f1aac064f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:47.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/2380312289 shutdown_connections 2023-06-23T01:07:47.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.375+0000 7f1ab2b9d700 1 -- 172.21.15.46:0/2380312289 wait complete. 2023-06-23T01:07:47.391 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-23T01:04:00.299796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-23T01:07:47.391 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T01:07:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.518+0000 7fe12fed9700 1 -- 172.21.15.46:0/1917947440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1280fce30 msgr2=0x7fe1280fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.518+0000 7fe12fed9700 1 --2- 172.21.15.46:0/1917947440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1280fce30 0x7fe1280fd280 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe1180049c0 tx=0x7fe118009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.518+0000 7fe12fed9700 1 -- 172.21.15.46:0/1917947440 shutdown_connections 2023-06-23T01:07:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.518+0000 7fe12fed9700 1 --2- 172.21.15.46:0/1917947440 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1280fd7c0 0x7fe128104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.518+0000 7fe12fed9700 1 --2- 172.21.15.46:0/1917947440 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1280fbc20 0x7fe1280fc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.518+0000 7fe12fed9700 1 --2- 172.21.15.46:0/1917947440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1280fce30 0x7fe1280fd280 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.518+0000 7fe12fed9700 1 -- 172.21.15.46:0/1917947440 >> 172.21.15.46:0/1917947440 conn(0x7fe1280f7380 msgr2=0x7fe1280f97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.518+0000 7fe12fed9700 1 -- 172.21.15.46:0/1917947440 shutdown_connections 2023-06-23T01:07:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.518+0000 7fe12fed9700 1 -- 172.21.15.46:0/1917947440 wait complete. 2023-06-23T01:07:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.519+0000 7fe12fed9700 1 Processor -- start 2023-06-23T01:07:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.519+0000 7fe12fed9700 1 -- start start 2023-06-23T01:07:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.519+0000 7fe12fed9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1280fbc20 0x7fe12810d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.519+0000 7fe12fed9700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1280fce30 0x7fe12810dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.519+0000 7fe12fed9700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1280fd7c0 0x7fe128114d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.519+0000 7fe12fed9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe128115360 con 0x7fe1280fbc20 2023-06-23T01:07:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.519+0000 7fe12fed9700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1281154d0 con 0x7fe1280fd7c0 2023-06-23T01:07:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.519+0000 7fe12fed9700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe128115690 con 0x7fe1280fce30 2023-06-23T01:07:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12dc75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1280fbc20 0x7fe12810d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12dc75700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1280fbc20 0x7fe12810d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60440/0 (socket says 172.21.15.46:60440) 2023-06-23T01:07:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12dc75700 1 -- 172.21.15.46:0/3958088359 learned_addr learned my addr 172.21.15.46:0/3958088359 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:47.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12e476700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1280fd7c0 0x7fe128114d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12d474700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1280fce30 0x7fe12810dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12e476700 1 -- 172.21.15.46:0/3958088359 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1280fce30 msgr2=0x7fe12810dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12e476700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1280fce30 0x7fe12810dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12e476700 1 -- 172.21.15.46:0/3958088359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1280fbc20 msgr2=0x7fe12810d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12e476700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1280fbc20 0x7fe12810d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12e476700 1 -- 172.21.15.46:0/3958088359 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe118005040 con 0x7fe1280fd7c0 2023-06-23T01:07:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12dc75700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1280fbc20 0x7fe12810d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:07:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12e476700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1280fd7c0 0x7fe128114d50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe12000ec20 tx=0x7fe12000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe11e7fc700 1 -- 172.21.15.46:0/3958088359 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe120007b30 con 0x7fe1280fd7c0 2023-06-23T01:07:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12fed9700 1 -- 172.21.15.46:0/3958088359 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe128115970 con 0x7fe1280fd7c0 2023-06-23T01:07:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe11e7fc700 1 -- 172.21.15.46:0/3958088359 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe12000f040 con 0x7fe1280fd7c0 2023-06-23T01:07:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe11e7fc700 1 -- 172.21.15.46:0/3958088359 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12000c920 con 0x7fe1280fd7c0 2023-06-23T01:07:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.520+0000 7fe12fed9700 1 -- 172.21.15.46:0/3958088359 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe128115ee0 con 0x7fe1280fd7c0 2023-06-23T01:07:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.521+0000 7fe12fed9700 1 -- 172.21.15.46:0/3958088359 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe128108430 con 0x7fe1280fd7c0 2023-06-23T01:07:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.522+0000 7fe11e7fc700 1 -- 172.21.15.46:0/3958088359 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fe12000cd10 con 0x7fe1280fd7c0 2023-06-23T01:07:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.523+0000 7fe11e7fc700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe11409b730 0x7fe11409dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.523+0000 7fe11e7fc700 1 -- 172.21.15.46:0/3958088359 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fe1200c4c60 con 0x7fe1280fd7c0 2023-06-23T01:07:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.523+0000 7fe12dc75700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe11409b730 0x7fe11409dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.523+0000 7fe12dc75700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe11409b730 0x7fe11409dbe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe124009fd0 tx=0x7fe12400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:47.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.526+0000 7fe11e7fc700 1 -- 172.21.15.46:0/3958088359 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fe120090960 con 0x7fe1280fd7c0 2023-06-23T01:07:47.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:47 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3163789047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:47.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:47 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2380312289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:47.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.727+0000 7fe12fed9700 1 -- 172.21.15.46:0/3958088359 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe128116070 con 0x7fe1280fd7c0 2023-06-23T01:07:47.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.727+0000 7fe11e7fc700 1 -- 172.21.15.46:0/3958088359 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe120093f80 con 0x7fe1280fd7c0 2023-06-23T01:07:47.730 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-06-23T01:07:47.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 -- 172.21.15.46:0/3958088359 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe11409b730 msgr2=0x7fe11409dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe11409b730 0x7fe11409dbe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe124009fd0 tx=0x7fe12400b040 comp rx=0 tx=0).stop 2023-06-23T01:07:47.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 -- 172.21.15.46:0/3958088359 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1280fd7c0 msgr2=0x7fe128114d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1280fd7c0 0x7fe128114d50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe12000ec20 tx=0x7fe12000b040 comp rx=0 tx=0).stop 2023-06-23T01:07:47.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 -- 172.21.15.46:0/3958088359 shutdown_connections 2023-06-23T01:07:47.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fe11409b730 0x7fe11409dbe0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe1280fd7c0 0x7fe128114d50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1280fbc20 0x7fe12810d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 --2- 172.21.15.46:0/3958088359 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fe1280fce30 0x7fe12810dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 -- 172.21.15.46:0/3958088359 >> 172.21.15.46:0/3958088359 conn(0x7fe1280f7380 msgr2=0x7fe1280f9510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:47.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 -- 172.21.15.46:0/3958088359 shutdown_connections 2023-06-23T01:07:47.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.729+0000 7fe12fed9700 1 -- 172.21.15.46:0/3958088359 wait complete. 2023-06-23T01:07:47.744 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-23T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.866+0000 7f34ec86d700 1 -- 172.21.15.46:0/2135820513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34e8070300 msgr2=0x7f34e8070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.866+0000 7f34ec86d700 1 --2- 172.21.15.46:0/2135820513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34e8070300 0x7f34e8070770 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f34d80049c0 tx=0x7f34d8009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:47.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.866+0000 7f34ec86d700 1 -- 172.21.15.46:0/2135820513 shutdown_connections 2023-06-23T01:07:47.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 --2- 172.21.15.46:0/2135820513 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34e8104a00 0x7f34e8106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 --2- 172.21.15.46:0/2135820513 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34e8071950 0x7f34e806fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 --2- 172.21.15.46:0/2135820513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34e8070300 0x7f34e8070770 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 -- 172.21.15.46:0/2135820513 >> 172.21.15.46:0/2135820513 conn(0x7f34e80f8260 msgr2=0x7f34e80fa690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:47.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 -- 172.21.15.46:0/2135820513 shutdown_connections 2023-06-23T01:07:47.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 -- 172.21.15.46:0/2135820513 wait complete. 2023-06-23T01:07:47.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 Processor -- start 2023-06-23T01:07:47.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 -- start start 2023-06-23T01:07:47.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34e8070300 0x7f34e810f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34e8071950 0x7f34e810fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34e8104a00 0x7f34e8116df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8117400 con 0x7f34e8071950 2023-06-23T01:07:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8117570 con 0x7f34e8070300 2023-06-23T01:07:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.867+0000 7f34ec86d700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8117730 con 0x7f34e8104a00 2023-06-23T01:07:47.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34e8071950 0x7f34e810fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e6d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34e8070300 0x7f34e810f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34e8071950 0x7f34e810fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60452/0 (socket says 172.21.15.46:60452) 2023-06-23T01:07:47.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e6d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34e8070300 0x7f34e810f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.46:55524/0 (socket says 172.21.15.46:55524) 2023-06-23T01:07:47.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e659c700 1 -- 172.21.15.46:0/735500739 learned_addr learned my addr 172.21.15.46:0/735500739 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e759e700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34e8104a00 0x7f34e8116df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e659c700 1 -- 172.21.15.46:0/735500739 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34e8104a00 msgr2=0x7f34e8116df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e659c700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34e8104a00 0x7f34e8116df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e659c700 1 -- 172.21.15.46:0/735500739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34e8070300 msgr2=0x7f34e810f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e659c700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34e8070300 0x7f34e810f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:47.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e659c700 1 -- 172.21.15.46:0/735500739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d8005040 con 0x7f34e8071950 2023-06-23T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34e659c700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34e8071950 0x7f34e810fd60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009c90 tx=0x7f34d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34cf7fe700 1 -- 172.21.15.46:0/735500739 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d800e040 con 0x7f34e8071950 2023-06-23T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34cf7fe700 1 -- 172.21.15.46:0/735500739 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f34d8009570 con 0x7f34e8071950 2023-06-23T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34ec86d700 1 -- 172.21.15.46:0/735500739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e81179b0 con 0x7f34e8071950 2023-06-23T01:07:47.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34cf7fe700 1 -- 172.21.15.46:0/735500739 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8013d60 con 0x7f34e8071950 2023-06-23T01:07:47.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.868+0000 7f34ec86d700 1 -- 172.21.15.46:0/735500739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e8117e90 con 0x7f34e8071950 2023-06-23T01:07:47.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.869+0000 7f34ec86d700 1 -- 172.21.15.46:0/735500739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e8103150 con 0x7f34e8071950 2023-06-23T01:07:47.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.870+0000 7f34cf7fe700 1 -- 172.21.15.46:0/735500739 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f34d8013520 con 0x7f34e8071950 2023-06-23T01:07:47.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.871+0000 7f34cf7fe700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f34d009b680 0x7f34d009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:47.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.871+0000 7f34e6d9d700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f34d009b680 0x7f34d009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:47.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.871+0000 7f34cf7fe700 1 -- 172.21.15.46:0/735500739 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f34d80c5a20 con 0x7f34e8071950 2023-06-23T01:07:47.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.872+0000 7f34e6d9d700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f34d009b680 0x7f34d009db30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f34dc004f40 tx=0x7f34dc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:47.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:47.876+0000 7f34cf7fe700 1 -- 172.21.15.46:0/735500739 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f34d80917d0 con 0x7f34e8071950 2023-06-23T01:07:47.981 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:47 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/3163789047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:47.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:47 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2380312289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:48.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.081+0000 7f34ec86d700 1 -- 172.21.15.46:0/735500739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f34e802b210 con 0x7f34e8071950 2023-06-23T01:07:48.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.081+0000 7f34cf7fe700 1 -- 172.21.15.46:0/735500739 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7f34d8094df0 con 0x7f34e8071950 2023-06-23T01:07:48.085 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:48.085 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2023-06-23T01:07:48.085 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-06-23T01:07:48.086 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2023-06-23T01:07:48.086 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2023-06-23T01:07:48.086 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-06-23T01:07:48.086 INFO:teuthology.orchestra.run.smithi046.stdout:] 2023-06-23T01:07:48.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 -- 172.21.15.46:0/735500739 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f34d009b680 msgr2=0x7f34d009db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:48.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f34d009b680 0x7f34d009db30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f34dc004f40 tx=0x7f34dc00b040 comp rx=0 tx=0).stop 2023-06-23T01:07:48.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 -- 172.21.15.46:0/735500739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34e8071950 msgr2=0x7f34e810fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:48.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34e8071950 0x7f34e810fd60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009c90 tx=0x7f34d8009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 -- 172.21.15.46:0/735500739 shutdown_connections 2023-06-23T01:07:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f34d009b680 0x7f34d009db30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f34e8070300 0x7f34e810f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f34e8104a00 0x7f34e8116df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 --2- 172.21.15.46:0/735500739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34e8071950 0x7f34e810fd60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 -- 172.21.15.46:0/735500739 >> 172.21.15.46:0/735500739 conn(0x7f34e80f8260 msgr2=0x7f34e80fa640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 -- 172.21.15.46:0/735500739 shutdown_connections 2023-06-23T01:07:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.083+0000 7f34ec86d700 1 -- 172.21.15.46:0/735500739 wait complete. 2023-06-23T01:07:48.099 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-23T01:07:48.100 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-23T01:07:48.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:47 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/3163789047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-06-23T01:07:48.139 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:47 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2380312289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.225+0000 7f03fac76700 1 -- 172.21.15.46:0/2574773706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03f40fd580 msgr2=0x7f03f4103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.225+0000 7f03fac76700 1 --2- 172.21.15.46:0/2574773706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03f40fd580 0x7f03f4103e40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009d60 tx=0x7f03ec004d10 comp rx=0 tx=0).stop 2023-06-23T01:07:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.225+0000 7f03fac76700 1 -- 172.21.15.46:0/2574773706 shutdown_connections 2023-06-23T01:07:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.225+0000 7f03fac76700 1 --2- 172.21.15.46:0/2574773706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03f40fb9e0 0x7f03f40fbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.225+0000 7f03fac76700 1 --2- 172.21.15.46:0/2574773706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03f40fd580 0x7f03f4103e40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.225+0000 7f03fac76700 1 --2- 172.21.15.46:0/2574773706 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f40fcbf0 0x7f03f40fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.225+0000 7f03fac76700 1 -- 172.21.15.46:0/2574773706 >> 172.21.15.46:0/2574773706 conn(0x7f03f40f7140 msgr2=0x7f03f40f9570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.225+0000 7f03fac76700 1 -- 172.21.15.46:0/2574773706 shutdown_connections 2023-06-23T01:07:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.225+0000 7f03fac76700 1 -- 172.21.15.46:0/2574773706 wait complete. 2023-06-23T01:07:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03fac76700 1 Processor -- start 2023-06-23T01:07:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03fac76700 1 -- start start 2023-06-23T01:07:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03fac76700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f40fb9e0 0x7f03f410d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03fac76700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03f40fcbf0 0x7f03f410daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03fac76700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03f40fd580 0x7f03f4114b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03fac76700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f4115190 con 0x7f03f40fcbf0 2023-06-23T01:07:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03fac76700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f4115300 con 0x7f03f40fd580 2023-06-23T01:07:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03fac76700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f41154c0 con 0x7f03f40fb9e0 2023-06-23T01:07:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03f9473700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03f40fcbf0 0x7f03f410daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03f9473700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03f40fcbf0 0x7f03f410daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60476/0 (socket says 172.21.15.46:60476) 2023-06-23T01:07:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03f9473700 1 -- 172.21.15.46:0/2411528043 learned_addr learned my addr 172.21.15.46:0/2411528043 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03f9c74700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f40fb9e0 0x7f03f410d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03fa475700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03f40fd580 0x7f03f4114b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03f9473700 1 -- 172.21.15.46:0/2411528043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f40fb9e0 msgr2=0x7f03f410d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03f9473700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f40fb9e0 0x7f03f410d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03f9473700 1 -- 172.21.15.46:0/2411528043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03f40fd580 msgr2=0x7f03f4114b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03f9473700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03f40fd580 0x7f03f4114b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.226+0000 7f03f9473700 1 -- 172.21.15.46:0/2411528043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f0005040 con 0x7f03f40fcbf0 2023-06-23T01:07:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.227+0000 7f03f9473700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03f40fcbf0 0x7f03f410daf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f03e400ebf0 tx=0x7f03e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.227+0000 7f03ea7fc700 1 -- 172.21.15.46:0/2411528043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e40047d0 con 0x7f03f40fcbf0 2023-06-23T01:07:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.227+0000 7f03fac76700 1 -- 172.21.15.46:0/2411528043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ec009940 con 0x7f03f40fcbf0 2023-06-23T01:07:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.227+0000 7f03ea7fc700 1 -- 172.21.15.46:0/2411528043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f03e400f040 con 0x7f03f40fcbf0 2023-06-23T01:07:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.227+0000 7f03ea7fc700 1 -- 172.21.15.46:0/2411528043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e400c820 con 0x7f03f40fcbf0 2023-06-23T01:07:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.227+0000 7f03fac76700 1 -- 172.21.15.46:0/2411528043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f4115ad0 con 0x7f03f40fcbf0 2023-06-23T01:07:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.228+0000 7f03fac76700 1 -- 172.21.15.46:0/2411528043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f4103910 con 0x7f03f40fcbf0 2023-06-23T01:07:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.229+0000 7f03ea7fc700 1 -- 172.21.15.46:0/2411528043 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f03e400cc50 con 0x7f03f40fcbf0 2023-06-23T01:07:48.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.230+0000 7f03ea7fc700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03e009b680 0x7f03e009db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:48.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.230+0000 7f03ea7fc700 1 -- 172.21.15.46:0/2411528043 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f03e40c4530 con 0x7f03f40fcbf0 2023-06-23T01:07:48.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.230+0000 7f03f9c74700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03e009b680 0x7f03e009db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:48.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.230+0000 7f03f9c74700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03e009b680 0x7f03e009db30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f03f40fca50 tx=0x7f03f000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:48.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.236+0000 7f03ea7fc700 1 -- 172.21.15.46:0/2411528043 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f03e4090230 con 0x7f03f40fcbf0 2023-06-23T01:07:48.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.455+0000 7f03fac76700 1 -- 172.21.15.46:0/2411528043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f03f402b180 con 0x7f03f40fcbf0 2023-06-23T01:07:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.936+0000 7f03ea7fc700 1 -- 172.21.15.46:0/2411528043 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7f03e4093850 con 0x7f03f40fcbf0 2023-06-23T01:07:48.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 -- 172.21.15.46:0/2411528043 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03e009b680 msgr2=0x7f03e009db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:48.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03e009b680 0x7f03e009db30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f03f40fca50 tx=0x7f03f000b040 comp rx=0 tx=0).stop 2023-06-23T01:07:48.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 -- 172.21.15.46:0/2411528043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03f40fcbf0 msgr2=0x7f03f410daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:48.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03f40fcbf0 0x7f03f410daf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f03e400ebf0 tx=0x7f03e400b040 comp rx=0 tx=0).stop 2023-06-23T01:07:48.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 -- 172.21.15.46:0/2411528043 shutdown_connections 2023-06-23T01:07:48.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f03e009b680 0x7f03e009db30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f03f40fd580 0x7f03f4114b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f03f40fb9e0 0x7f03f410d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 --2- 172.21.15.46:0/2411528043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03f40fcbf0 0x7f03f410daf0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:48.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 -- 172.21.15.46:0/2411528043 >> 172.21.15.46:0/2411528043 conn(0x7f03f40f7140 msgr2=0x7f03f40f9250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:48.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 -- 172.21.15.46:0/2411528043 shutdown_connections 2023-06-23T01:07:48.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:48.942+0000 7f03fac76700 1 -- 172.21.15.46:0/2411528043 wait complete. 2023-06-23T01:07:48.949 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2023-06-23T01:07:48.959 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-23T01:07:49.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.088+0000 7f7a19c9c700 1 -- 172.21.15.46:0/2923163581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a14066720 msgr2=0x7f7a140ff6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:49.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.088+0000 7f7a19c9c700 1 --2- 172.21.15.46:0/2923163581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a14066720 0x7f7a140ff6a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7a08004900 tx=0x7f7a08009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:49.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.088+0000 7f7a19c9c700 1 -- 172.21.15.46:0/2923163581 shutdown_connections 2023-06-23T01:07:49.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.088+0000 7f7a19c9c700 1 --2- 172.21.15.46:0/2923163581 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a14066720 0x7f7a140ff6a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.088+0000 7f7a19c9c700 1 --2- 172.21.15.46:0/2923163581 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a14102760 0x7f7a14104c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.088+0000 7f7a19c9c700 1 --2- 172.21.15.46:0/2923163581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a140ffc70 0x7f7a141020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.088+0000 7f7a19c9c700 1 -- 172.21.15.46:0/2923163581 >> 172.21.15.46:0/2923163581 conn(0x7f7a140f7130 msgr2=0x7f7a140f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.088+0000 7f7a19c9c700 1 -- 172.21.15.46:0/2923163581 shutdown_connections 2023-06-23T01:07:49.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.088+0000 7f7a19c9c700 1 -- 172.21.15.46:0/2923163581 wait complete. 2023-06-23T01:07:49.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a19c9c700 1 Processor -- start 2023-06-23T01:07:49.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a19c9c700 1 -- start start 2023-06-23T01:07:49.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a19c9c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a14066720 0x7f7a1410d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:49.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a19c9c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a140ffc70 0x7f7a1410dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:49.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a19c9c700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a14102760 0x7f7a14114c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:49.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a19c9c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a14115280 con 0x7f7a140ffc70 2023-06-23T01:07:49.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a19c9c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a141153f0 con 0x7f7a14066720 2023-06-23T01:07:49.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a19c9c700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a141155b0 con 0x7f7a14102760 2023-06-23T01:07:49.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a13fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a140ffc70 0x7f7a1410dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:49.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a13fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a140ffc70 0x7f7a1410dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60496/0 (socket says 172.21.15.46:60496) 2023-06-23T01:07:49.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a13fff700 1 -- 172.21.15.46:0/4115996392 learned_addr learned my addr 172.21.15.46:0/4115996392 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:49.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a18c9a700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a14066720 0x7f7a1410d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:49.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a1949b700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a14102760 0x7f7a14114c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:49.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a18c9a700 1 -- 172.21.15.46:0/4115996392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a14102760 msgr2=0x7f7a14114c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a18c9a700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a14102760 0x7f7a14114c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a18c9a700 1 -- 172.21.15.46:0/4115996392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a140ffc70 msgr2=0x7f7a1410dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a18c9a700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a140ffc70 0x7f7a1410dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a18c9a700 1 -- 172.21.15.46:0/4115996392 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a00005040 con 0x7f7a14066720 2023-06-23T01:07:49.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a13fff700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a140ffc70 0x7f7a1410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:07:49.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.089+0000 7f7a1949b700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a14102760 0x7f7a14114c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:07:49.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.090+0000 7f7a18c9a700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a14066720 0x7f7a1410d6a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a080047d0 tx=0x7f7a08009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.090+0000 7f7a117fa700 1 -- 172.21.15.46:0/4115996392 <== mon.2 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0800e040 con 0x7f7a14066720 2023-06-23T01:07:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.090+0000 7f7a117fa700 1 -- 172.21.15.46:0/4115996392 <== mon.2 v2:172.21.15.92:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7a08009710 con 0x7f7a14066720 2023-06-23T01:07:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.090+0000 7f7a19c9c700 1 -- 172.21.15.46:0/4115996392 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a0c0048f0 con 0x7f7a14066720 2023-06-23T01:07:49.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.090+0000 7f7a117fa700 1 -- 172.21.15.46:0/4115996392 <== mon.2 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0801cd10 con 0x7f7a14066720 2023-06-23T01:07:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.090+0000 7f7a19c9c700 1 -- 172.21.15.46:0/4115996392 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a08005040 con 0x7f7a14066720 2023-06-23T01:07:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.091+0000 7f7a19c9c700 1 -- 172.21.15.46:0/4115996392 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a140f8c60 con 0x7f7a14066720 2023-06-23T01:07:49.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.092+0000 7f7a117fa700 1 -- 172.21.15.46:0/4115996392 <== mon.2 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f7a0801c430 con 0x7f7a14066720 2023-06-23T01:07:49.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.092+0000 7f7a117fa700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a0409b680 0x7f7a0409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:49.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.093+0000 7f7a13fff700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a0409b680 0x7f7a0409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:49.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.093+0000 7f7a117fa700 1 -- 172.21.15.46:0/4115996392 <== mon.2 v2:172.21.15.92:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f7a080c6160 con 0x7f7a14066720 2023-06-23T01:07:49.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.093+0000 7f7a13fff700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a0409b680 0x7f7a0409db30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009170 tx=0x7f7a0000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:49.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.096+0000 7f7a117fa700 1 -- 172.21.15.46:0/4115996392 <== mon.2 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f7a08091cf0 con 0x7f7a14066720 2023-06-23T01:07:49.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:48 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/3958088359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-23T01:07:49.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:48 smithi131 ceph-mon[86495]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:49.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:48 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/735500739' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-06-23T01:07:49.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:48 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2411528043' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-06-23T01:07:49.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.296+0000 7f7a19c9c700 1 -- 172.21.15.46:0/4115996392 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7a141081b0 con 0x7f7a14066720 2023-06-23T01:07:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:48 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3958088359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-23T01:07:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:48 smithi046 ceph-mon[84949]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:48 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/735500739' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-06-23T01:07:49.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:48 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2411528043' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-06-23T01:07:49.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:48 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/3958088359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-06-23T01:07:49.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:48 smithi092 ceph-mon[85670]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:49.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:48 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/735500739' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-06-23T01:07:49.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:48 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2411528043' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-06-23T01:07:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.946+0000 7f7a117fa700 1 -- 172.21.15.46:0/4115996392 <== mon.2 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7a08095310 con 0x7f7a14066720 2023-06-23T01:07:49.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.951+0000 7f7a19c9c700 1 -- 172.21.15.46:0/4115996392 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a0409b680 msgr2=0x7f7a0409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.951+0000 7f7a19c9c700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a0409b680 0x7f7a0409db30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009170 tx=0x7f7a0000b040 comp rx=0 tx=0).stop 2023-06-23T01:07:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.951+0000 7f7a19c9c700 1 -- 172.21.15.46:0/4115996392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a14066720 msgr2=0x7f7a1410d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:49.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.951+0000 7f7a19c9c700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a14066720 0x7f7a1410d6a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7a080047d0 tx=0x7f7a08009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.953+0000 7f7a19c9c700 1 -- 172.21.15.46:0/4115996392 shutdown_connections 2023-06-23T01:07:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.953+0000 7f7a19c9c700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f7a0409b680 0x7f7a0409db30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.953+0000 7f7a19c9c700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7a14066720 0x7f7a1410d6a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:49.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.953+0000 7f7a19c9c700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f7a14102760 0x7f7a14114c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.953+0000 7f7a19c9c700 1 --2- 172.21.15.46:0/4115996392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a140ffc70 0x7f7a1410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.953+0000 7f7a19c9c700 1 -- 172.21.15.46:0/4115996392 >> 172.21.15.46:0/4115996392 conn(0x7f7a140f7130 msgr2=0x7f7a140fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.954+0000 7f7a19c9c700 1 -- 172.21.15.46:0/4115996392 shutdown_connections 2023-06-23T01:07:49.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:49.954+0000 7f7a19c9c700 1 -- 172.21.15.46:0/4115996392 wait complete. 2023-06-23T01:07:49.964 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2023-06-23T01:07:49.982 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-23T01:07:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.107+0000 7f4801ec5700 1 -- 172.21.15.46:0/3187377563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc0fed70 msgr2=0x7f47fc0fae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.107+0000 7f4801ec5700 1 --2- 172.21.15.46:0/3187377563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc0fed70 0x7f47fc0fae20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f47e40049c0 tx=0x7f47e4009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.107+0000 7f4801ec5700 1 -- 172.21.15.46:0/3187377563 shutdown_connections 2023-06-23T01:07:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.107+0000 7f4801ec5700 1 --2- 172.21.15.46:0/3187377563 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47fc0fb490 0x7f47fc105390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.107+0000 7f4801ec5700 1 --2- 172.21.15.46:0/3187377563 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47fc0fe390 0x7f47fc0fe7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.107+0000 7f4801ec5700 1 --2- 172.21.15.46:0/3187377563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc0fed70 0x7f47fc0fae20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.107+0000 7f4801ec5700 1 -- 172.21.15.46:0/3187377563 >> 172.21.15.46:0/3187377563 conn(0x7f47fc0f6b30 msgr2=0x7f47fc0f8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.107+0000 7f4801ec5700 1 -- 172.21.15.46:0/3187377563 shutdown_connections 2023-06-23T01:07:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.107+0000 7f4801ec5700 1 -- 172.21.15.46:0/3187377563 wait complete. 2023-06-23T01:07:50.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.108+0000 7f4801ec5700 1 Processor -- start 2023-06-23T01:07:50.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.108+0000 7f4801ec5700 1 -- start start 2023-06-23T01:07:50.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.108+0000 7f4801ec5700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47fc0fb490 0x7f47fc10d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f4801ec5700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47fc0fe390 0x7f47fc10dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f4801ec5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc112df0 0x7f47fc1151d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f4801ec5700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc1157e0 con 0x7f47fc112df0 2023-06-23T01:07:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f4801ec5700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc115950 con 0x7f47fc0fe390 2023-06-23T01:07:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f4801ec5700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc115b10 con 0x7f47fc0fb490 2023-06-23T01:07:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47fbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc112df0 0x7f47fc1151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47fbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc112df0 0x7f47fc1151d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60520/0 (socket says 172.21.15.46:60520) 2023-06-23T01:07:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47fbfff700 1 -- 172.21.15.46:0/3505635919 learned_addr learned my addr 172.21.15.46:0/3505635919 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47fb7fe700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47fc0fb490 0x7f47fc10d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47faffd700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47fc0fe390 0x7f47fc10dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47fbfff700 1 -- 172.21.15.46:0/3505635919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47fc0fb490 msgr2=0x7f47fc10d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47fbfff700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47fc0fb490 0x7f47fc10d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47fbfff700 1 -- 172.21.15.46:0/3505635919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47fc0fe390 msgr2=0x7f47fc10dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47fbfff700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47fc0fe390 0x7f47fc10dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47fbfff700 1 -- 172.21.15.46:0/3505635919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e4005040 con 0x7f47fc112df0 2023-06-23T01:07:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47fbfff700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc112df0 0x7f47fc1151d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f47f400ec30 tx=0x7f47f400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47f3fff700 1 -- 172.21.15.46:0/3505635919 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f4007b30 con 0x7f47fc112df0 2023-06-23T01:07:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47f3fff700 1 -- 172.21.15.46:0/3505635919 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f47f400f040 con 0x7f47fc112df0 2023-06-23T01:07:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f47f3fff700 1 -- 172.21.15.46:0/3505635919 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f400c9c0 con 0x7f47fc112df0 2023-06-23T01:07:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f4801ec5700 1 -- 172.21.15.46:0/3505635919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47fc115df0 con 0x7f47fc112df0 2023-06-23T01:07:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.109+0000 7f4801ec5700 1 -- 172.21.15.46:0/3505635919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47fc116280 con 0x7f47fc112df0 2023-06-23T01:07:50.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.110+0000 7f4801ec5700 1 -- 172.21.15.46:0/3505635919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47fc108450 con 0x7f47fc112df0 2023-06-23T01:07:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.112+0000 7f47f3fff700 1 -- 172.21.15.46:0/3505635919 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f47f4009090 con 0x7f47fc112df0 2023-06-23T01:07:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.112+0000 7f47f3fff700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f47e809b730 0x7f47e809dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.112+0000 7f47fb7fe700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f47e809b730 0x7f47e809dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.112+0000 7f47f3fff700 1 -- 172.21.15.46:0/3505635919 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7875+0+0 (secure 0 0 0) 0x7f47f40c5650 con 0x7f47fc112df0 2023-06-23T01:07:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.113+0000 7f47fb7fe700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f47e809b730 0x7f47e809dbe0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f47fc064c30 tx=0x7f47ec00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:50.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.117+0000 7f47f3fff700 1 -- 172.21.15.46:0/3505635919 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f47f4091060 con 0x7f47fc112df0 2023-06-23T01:07:50.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/2411528043' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-06-23T01:07:50.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: osdmap e78: 12 total, 12 up, 12 in 2023-06-23T01:07:50.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/4115996392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-06-23T01:07:50.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-06-23T01:07:50.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:50.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:50.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:49 smithi131 ceph-mon[86495]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:50.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/2411528043' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-06-23T01:07:50.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: osdmap e78: 12 total, 12 up, 12 in 2023-06-23T01:07:50.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/4115996392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-06-23T01:07:50.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-06-23T01:07:50.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:50.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:50.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.320 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:49 smithi046 ceph-mon[84949]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:50.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.367+0000 7f4801ec5700 1 -- 172.21.15.46:0/3505635919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f47fc0fee60 con 0x7f47fc112df0 2023-06-23T01:07:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.377+0000 7f47f3fff700 1 -- 172.21.15.46:0/3505635919 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f47f4094680 con 0x7f47fc112df0 2023-06-23T01:07:50.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/2411528043' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-06-23T01:07:50.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: osdmap e78: 12 total, 12 up, 12 in 2023-06-23T01:07:50.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/4115996392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-06-23T01:07:50.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-06-23T01:07:50.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-06-23T01:07:50.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-06-23T01:07:50.392 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' 2023-06-23T01:07:50.393 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:49 smithi092 ceph-mon[85670]: from='mgr.14152 172.21.15.46:0/2094119537' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-06-23T01:07:50.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.398+0000 7f4801ec5700 1 -- 172.21.15.46:0/3505635919 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f47e809b730 msgr2=0x7f47e809dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.398+0000 7f4801ec5700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f47e809b730 0x7f47e809dbe0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f47fc064c30 tx=0x7f47ec00b040 comp rx=0 tx=0).stop 2023-06-23T01:07:50.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.398+0000 7f4801ec5700 1 -- 172.21.15.46:0/3505635919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc112df0 msgr2=0x7f47fc1151d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.398+0000 7f4801ec5700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc112df0 0x7f47fc1151d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f47f400ec30 tx=0x7f47f400b040 comp rx=0 tx=0).stop 2023-06-23T01:07:50.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.401+0000 7f4801ec5700 1 -- 172.21.15.46:0/3505635919 shutdown_connections 2023-06-23T01:07:50.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.401+0000 7f4801ec5700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc112df0 0x7f47fc1151d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.401+0000 7f4801ec5700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f47fc0fb490 0x7f47fc10d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.401+0000 7f4801ec5700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f47e809b730 0x7f47e809dbe0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.401+0000 7f4801ec5700 1 --2- 172.21.15.46:0/3505635919 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f47fc0fe390 0x7f47fc10dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.401+0000 7f4801ec5700 1 -- 172.21.15.46:0/3505635919 >> 172.21.15.46:0/3505635919 conn(0x7f47fc0f6b30 msgr2=0x7f47fc0f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:50.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.402+0000 7f4801ec5700 1 -- 172.21.15.46:0/3505635919 shutdown_connections 2023-06-23T01:07:50.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.402+0000 7f4801ec5700 1 -- 172.21.15.46:0/3505635919 wait complete. 2023-06-23T01:07:50.406 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 2 and data pool 3 2023-06-23T01:07:50.420 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:07:50.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.539+0000 7fbbd2a89700 1 -- 172.21.15.46:0/284597176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc070300 msgr2=0x7fbbcc070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.539+0000 7fbbd2a89700 1 --2- 172.21.15.46:0/284597176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc070300 0x7fbbcc070770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80049c0 tx=0x7fbbb8009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:50.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 -- 172.21.15.46:0/284597176 shutdown_connections 2023-06-23T01:07:50.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 --2- 172.21.15.46:0/284597176 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbcc104be0 0x7fbbcc107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 --2- 172.21.15.46:0/284597176 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbcc071950 0x7fbbcc06fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 --2- 172.21.15.46:0/284597176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc070300 0x7fbbcc070770 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 -- 172.21.15.46:0/284597176 >> 172.21.15.46:0/284597176 conn(0x7fbbcc0f8420 msgr2=0x7fbbcc0fa870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:50.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 -- 172.21.15.46:0/284597176 shutdown_connections 2023-06-23T01:07:50.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 -- 172.21.15.46:0/284597176 wait complete. 2023-06-23T01:07:50.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 Processor -- start 2023-06-23T01:07:50.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 -- start start 2023-06-23T01:07:50.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbcc070300 0x7fbbcc10fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc071950 0x7fbbcc10ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbcc104be0 0x7fbbcc116fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc1175e0 con 0x7fbbcc071950 2023-06-23T01:07:50.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc117750 con 0x7fbbcc104be0 2023-06-23T01:07:50.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.540+0000 7fbbd2a89700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc117910 con 0x7fbbcc070300 2023-06-23T01:07:50.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbcbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc071950 0x7fbbcc10ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbcbfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc071950 0x7fbbcc10ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60532/0 (socket says 172.21.15.46:60532) 2023-06-23T01:07:50.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbcbfff700 1 -- 172.21.15.46:0/191662845 learned_addr learned my addr 172.21.15.46:0/191662845 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:50.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbd1026700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbcc104be0 0x7fbbcc116fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbd0825700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbcc070300 0x7fbbcc10fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbcbfff700 1 -- 172.21.15.46:0/191662845 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbcc070300 msgr2=0x7fbbcc10fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbcbfff700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbcc070300 0x7fbbcc10fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbcbfff700 1 -- 172.21.15.46:0/191662845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbcc104be0 msgr2=0x7fbbcc116fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbcbfff700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbcc104be0 0x7fbbcc116fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbcbfff700 1 -- 172.21.15.46:0/191662845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8005040 con 0x7fbbcc071950 2023-06-23T01:07:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbd1026700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbcc104be0 0x7fbbcc116fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:07:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbcbfff700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc071950 0x7fbbcc10ff40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800b3b0 tx=0x7fbbb8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:50.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbc97fa700 1 -- 172.21.15.46:0/191662845 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb800e040 con 0x7fbbcc071950 2023-06-23T01:07:50.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbd2a89700 1 -- 172.21.15.46:0/191662845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbcc117b90 con 0x7fbbcc071950 2023-06-23T01:07:50.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbc97fa700 1 -- 172.21.15.46:0/191662845 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbbb8009640 con 0x7fbbcc071950 2023-06-23T01:07:50.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbc97fa700 1 -- 172.21.15.46:0/191662845 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8013cf0 con 0x7fbbcc071950 2023-06-23T01:07:50.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.541+0000 7fbbd2a89700 1 -- 172.21.15.46:0/191662845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbcc118040 con 0x7fbbcc071950 2023-06-23T01:07:50.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.543+0000 7fbbd2a89700 1 -- 172.21.15.46:0/191662845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbcc103330 con 0x7fbbcc071950 2023-06-23T01:07:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.543+0000 7fbbc97fa700 1 -- 172.21.15.46:0/191662845 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fbbb8013420 con 0x7fbbcc071950 2023-06-23T01:07:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.544+0000 7fbbc97fa700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fbbbc09b680 0x7fbbbc09db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.544+0000 7fbbc97fa700 1 -- 172.21.15.46:0/191662845 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fbbb80c5f50 con 0x7fbbcc071950 2023-06-23T01:07:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.544+0000 7fbbd0825700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fbbbc09b680 0x7fbbbc09db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.544+0000 7fbbd0825700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fbbbc09b680 0x7fbbbc09db30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0007fd0 tx=0x7fbbc0009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:50.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.548+0000 7fbbc97fa700 1 -- 172.21.15.46:0/191662845 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fbbb8091900 con 0x7fbbcc071950 2023-06-23T01:07:50.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.752+0000 7fbbd2a89700 1 -- 172.21.15.46:0/191662845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbcc10a710 con 0x7fbbcc071950 2023-06-23T01:07:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.753+0000 7fbbc97fa700 1 -- 172.21.15.46:0/191662845 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+21335 (secure 0 0 0) 0x7fbbb8094f20 con 0x7fbbcc071950 2023-06-23T01:07:50.755 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:50.756 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"fsid":"458642ee-1161-11ee-9b2b-001a4aab830c","created":"2023-06-23T01:00:57.877538+0000","modified":"2023-06-23T01:07:50.368379+0000","last_up_change":"2023-06-23T01:07:07.914869+0000","last_in_change":"2023-06-23T01:06:51.140023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:04:00.299796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-06-23T01:07:48.883010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-06-23T01:07:49.722003+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f28767ce-a780-4e50-b118-37a8c275c0e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6803","nonce":709285010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6805","nonce":709285010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6809","nonce":709285010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6807","nonce":709285010}]},"public_addr":"172.21.15.46:6803/709285010","cluster_addr":"172.21.15.46:6805/709285010","heartbeat_back_addr":"172.21.15.46:6809/709285010","heartbeat_front_addr":"172.21.15.46:6807/709285010","state":["exists","up"]},{"osd":1,"uuid":"4d4c4c36-d4a1-4b96-9059-adffee04401a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6801","nonce":4266072183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6803","nonce":4266072183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6807","nonce":4266072183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6805","nonce":4266072183}]},"public_addr":"172.21.15.92:6801/4266072183","cluster_addr":"172.21.15.92:6803/4266072183","heartbeat_back_addr":"172.21.15.92:6807/4266072183","heartbeat_front_addr":"172.21.15.92:6805/4266072183","state":["exists","up"]},{"osd":2,"uuid":"ec7b0c60-88ea-4c2b-a34c-96d188e02014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6801","nonce":1787957441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6803","nonce":1787957441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6807","nonce":1787957441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6805","nonce":1787957441}]},"public_addr":"172.21.15.131:6801/1787957441","cluster_addr":"172.21.15.131:6803/1787957441","heartbeat_back_addr":"172.21.15.131:6807/1787957441","heartbeat_front_addr":"172.21.15.131:6805/1787957441","state":["exists","up"]},{"osd":3,"uuid":"32670a23-3286-4538-9628-38d763429cbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6811","nonce":338690444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6813","nonce":338690444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6817","nonce":338690444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6815","nonce":338690444}]},"public_addr":"172.21.15.46:6811/338690444","cluster_addr":"172.21.15.46:6813/338690444","heartbeat_back_addr":"172.21.15.46:6817/338690444","heartbeat_front_addr":"172.21.15.46:6815/338690444","state":["exists","up"]},{"osd":4,"uuid":"0d8f982c-eb74-48a2-949e-d0ccd7732937","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6809","nonce":2024629902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6811","nonce":2024629902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6815","nonce":2024629902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6813","nonce":2024629902}]},"public_addr":"172.21.15.92:6809/2024629902","cluster_addr":"172.21.15.92:6811/2024629902","heartbeat_back_addr":"172.21.15.92:6815/2024629902","heartbeat_front_addr":"172.21.15.92:6813/2024629902","state":["exists","up"]},{"osd":5,"uuid":"52a7160c-4735-4a77-bc08-4d411c282b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6809","nonce":2577961971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6811","nonce":2577961971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6815","nonce":2577961971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6813","nonce":2577961971}]},"public_addr":"172.21.15.131:6809/2577961971","cluster_addr":"172.21.15.131:6811/2577961971","heartbeat_back_addr":"172.21.15.131:6815/2577961971","heartbeat_front_addr":"172.21.15.131:6813/2577961971","state":["exists","up"]},{"osd":6,"uuid":"3f3d2690-f655-4dda-a36c-bc97f9d08a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6819","nonce":558288790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6821","nonce":558288790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6825","nonce":558288790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6823","nonce":558288790}]},"public_addr":"172.21.15.46:6819/558288790","cluster_addr":"172.21.15.46:6821/558288790","heartbeat_back_addr":"172.21.15.46:6825/558288790","heartbeat_front_addr":"172.21.15.46:6823/558288790","state":["exists","up"]},{"osd":7,"uuid":"2eb43609-8921-4891-b0c4-af932bfb69bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6817","nonce":3796540277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6819","nonce":3796540277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6823","nonce":3796540277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6821","nonce":3796540277}]},"public_addr":"172.21.15.92:6817/3796540277","cluster_addr":"172.21.15.92:6819/3796540277","heartbeat_back_addr":"172.21.15.92:6823/3796540277","heartbeat_front_addr":"172.21.15.92:6821/3796540277","state":["exists","up"]},{"osd":8,"uuid":"0139b39e-f933-4685-b056-a6c27fa87fd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6817","nonce":1004740496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6819","nonce":1004740496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6823","nonce":1004740496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6821","nonce":1004740496}]},"public_addr":"172.21.15.131:6817/1004740496","cluster_addr":"172.21.15.131:6819/1004740496","heartbeat_back_addr":"172.21.15.131:6823/1004740496","heartbeat_front_addr":"172.21.15.131:6821/1004740496","state":["exists","up"]},{"osd":9,"uuid":"8bd083b4-f766-4e1b-b959-19d2fa82d77f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6827","nonce":3042170676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6829","nonce":3042170676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6833","nonce":3042170676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6831","nonce":3042170676}]},"public_addr":"172.21.15.46:6827/3042170676","cluster_addr":"172.21.15.46:6829/3042170676","heartbeat_back_addr":"172.21.15.46:6833/3042170676","heartbeat_front_addr":"172.21.15.46:6831/3042170676","state":["exists","up"]},{"osd":10,"uuid":"0b4f58f5-d971-4634-a1b6-896501d3d218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6825","nonce":2375910823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6827","nonce":2375910823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6831","nonce":2375910823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6829","nonce":2375910823}]},"public_addr":"172.21.15.92:6825/2375910823","cluster_addr":"172.21.15.92:6827/2375910823","heartbeat_back_addr":"172.21.15.92:6831/2375910823","heartbeat_front_addr":"172.21.15.92:6829/2375910823","state":["exists","up"]},{"osd":11,"uuid":"7d7a4115-c6e9-4cef-865b-05f57fbcface","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6825","nonce":1716414800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6827","nonce":1716414800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6831","nonce":1716414800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6829","nonce":1716414800}]},"public_addr":"172.21.15.131:6825/1716414800","cluster_addr":"172.21.15.131:6827/1716414800","heartbeat_back_addr":"172.21.15.131:6831/1716414800","heartbeat_front_addr":"172.21.15.131:6829/1716414800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:15.785940+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:36.479807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:57.678238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:04:20.396101+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:04:41.179881+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:01.570524+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:22.313074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:43.129445+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:03.897470+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:25.196569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:46.208577+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:07:06.788656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/685567953":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6801/2331157358":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6800/2331157358":"2023-06-24T01:01:55.786463+0000","172.21.15.46:0/1655911006":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/187730254":"2023-06-24T01:01:32.095580+0000","172.21.15.46:6801/3291477171":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/3127678959":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6800/3291477171":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/411746526":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/360533287":"2023-06-24T01:01:55.786463+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:07:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 -- 172.21.15.46:0/191662845 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fbbbc09b680 msgr2=0x7fbbbc09db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fbbbc09b680 0x7fbbbc09db30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0007fd0 tx=0x7fbbc0009040 comp rx=0 tx=0).stop 2023-06-23T01:07:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 -- 172.21.15.46:0/191662845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc071950 msgr2=0x7fbbcc10ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc071950 0x7fbbcc10ff40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800b3b0 tx=0x7fbbb8009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 -- 172.21.15.46:0/191662845 shutdown_connections 2023-06-23T01:07:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fbbbc09b680 0x7fbbbc09db30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbcc104be0 0x7fbbcc116fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fbbcc070300 0x7fbbcc10fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 --2- 172.21.15.46:0/191662845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbcc071950 0x7fbbcc10ff40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 -- 172.21.15.46:0/191662845 >> 172.21.15.46:0/191662845 conn(0x7fbbcc0f8420 msgr2=0x7fbbcc0fa800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 -- 172.21.15.46:0/191662845 shutdown_connections 2023-06-23T01:07:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.755+0000 7fbbd2a89700 1 -- 172.21.15.46:0/191662845 wait complete. 2023-06-23T01:07:50.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:07:50.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.892+0000 7f25dfa82700 1 -- 172.21.15.46:0/1212162615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d80fbbd0 msgr2=0x7f25d80fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.892+0000 7f25dfa82700 1 --2- 172.21.15.46:0/1212162615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d80fbbd0 0x7f25d80fbfe0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f25d40049c0 tx=0x7f25d4009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:50.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.892+0000 7f25dfa82700 1 -- 172.21.15.46:0/1212162615 shutdown_connections 2023-06-23T01:07:50.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.892+0000 7f25dfa82700 1 --2- 172.21.15.46:0/1212162615 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25d80fd770 0x7f25d8104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.892+0000 7f25dfa82700 1 --2- 172.21.15.46:0/1212162615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d80fbbd0 0x7f25d80fbfe0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.892+0000 7f25dfa82700 1 --2- 172.21.15.46:0/1212162615 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d80fcde0 0x7f25d80fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.892+0000 7f25dfa82700 1 -- 172.21.15.46:0/1212162615 >> 172.21.15.46:0/1212162615 conn(0x7f25d80f7310 msgr2=0x7f25d80f9760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:50.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.892+0000 7f25dfa82700 1 -- 172.21.15.46:0/1212162615 shutdown_connections 2023-06-23T01:07:50.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.892+0000 7f25dfa82700 1 -- 172.21.15.46:0/1212162615 wait complete. 2023-06-23T01:07:50.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25dfa82700 1 Processor -- start 2023-06-23T01:07:50.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25dfa82700 1 -- start start 2023-06-23T01:07:50.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25dfa82700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d80fbbd0 0x7f25d8065a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25dfa82700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25d80fcde0 0x7f25d8112630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25dfa82700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d80fd770 0x7f25d8114c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25dfa82700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d8115410 con 0x7f25d80fd770 2023-06-23T01:07:50.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25dfa82700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d8115580 con 0x7f25d80fcde0 2023-06-23T01:07:50.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25dfa82700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d8115740 con 0x7f25d80fbbd0 2023-06-23T01:07:50.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25de01f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d80fd770 0x7f25d8114c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25de01f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d80fd770 0x7f25d8114c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60550/0 (socket says 172.21.15.46:60550) 2023-06-23T01:07:50.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25dd01d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25d80fcde0 0x7f25d8112630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25de01f700 1 -- 172.21.15.46:0/3755172317 learned_addr learned my addr 172.21.15.46:0/3755172317 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:50.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.893+0000 7f25dd81e700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d80fbbd0 0x7f25d8065a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.894+0000 7f25de01f700 1 -- 172.21.15.46:0/3755172317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d80fbbd0 msgr2=0x7f25d8065a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.894+0000 7f25de01f700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d80fbbd0 0x7f25d8065a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.894+0000 7f25de01f700 1 -- 172.21.15.46:0/3755172317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25d80fcde0 msgr2=0x7f25d8112630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:50.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.894+0000 7f25de01f700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25d80fcde0 0x7f25d8112630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:50.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.894+0000 7f25de01f700 1 -- 172.21.15.46:0/3755172317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d4005040 con 0x7f25d80fd770 2023-06-23T01:07:50.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.894+0000 7f25de01f700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d80fd770 0x7f25d8114c70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f25d000ec30 tx=0x7f25d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:50.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.894+0000 7f25ce7fc700 1 -- 172.21.15.46:0/3755172317 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d0007b30 con 0x7f25d80fd770 2023-06-23T01:07:50.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.894+0000 7f25ce7fc700 1 -- 172.21.15.46:0/3755172317 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f25d000f040 con 0x7f25d80fd770 2023-06-23T01:07:50.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.894+0000 7f25dfa82700 1 -- 172.21.15.46:0/3755172317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d8115a20 con 0x7f25d80fd770 2023-06-23T01:07:50.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.894+0000 7f25ce7fc700 1 -- 172.21.15.46:0/3755172317 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d000c9c0 con 0x7f25d80fd770 2023-06-23T01:07:50.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.894+0000 7f25dfa82700 1 -- 172.21.15.46:0/3755172317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d8115f90 con 0x7f25d80fd770 2023-06-23T01:07:50.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.895+0000 7f25dfa82700 1 -- 172.21.15.46:0/3755172317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d8107270 con 0x7f25d80fd770 2023-06-23T01:07:50.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.897+0000 7f25ce7fc700 1 -- 172.21.15.46:0/3755172317 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f25d0009090 con 0x7f25d80fd770 2023-06-23T01:07:50.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.898+0000 7f25ce7fc700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f25c409b680 0x7f25c409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:50.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.898+0000 7f25ce7fc700 1 -- 172.21.15.46:0/3755172317 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f25d00c56f0 con 0x7f25d80fd770 2023-06-23T01:07:50.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.899+0000 7f25dd81e700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f25c409b680 0x7f25c409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:50.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.899+0000 7f25dd81e700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f25c409b680 0x7f25c409db30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009b80 tx=0x7f25d4009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:50.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:50.901+0000 7f25ce7fc700 1 -- 172.21.15.46:0/3755172317 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f25d00910a0 con 0x7f25d80fd770 2023-06-23T01:07:51.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:50 smithi046 ceph-mon[84949]: pgmap v245: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:51.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:50 smithi046 ceph-mon[84949]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-06-23T01:07:51.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:50 smithi046 ceph-mon[84949]: osdmap e79: 12 total, 12 up, 12 in 2023-06-23T01:07:51.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:50 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3505635919' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-06-23T01:07:51.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:50 smithi046 ceph-mon[84949]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-06-23T01:07:51.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:50 smithi046 ceph-mon[84949]: osdmap e80: 12 total, 12 up, 12 in 2023-06-23T01:07:51.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:50 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3505635919' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-06-23T01:07:51.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:50 smithi046 ceph-mon[84949]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-06-23T01:07:51.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:50 smithi046 ceph-mon[84949]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-06-23T01:07:51.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:50 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/191662845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:51.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.101+0000 7f25dfa82700 1 -- 172.21.15.46:0/3755172317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25d8116120 con 0x7f25d80fd770 2023-06-23T01:07:51.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.102+0000 7f25ce7fc700 1 -- 172.21.15.46:0/3755172317 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+21335 (secure 0 0 0) 0x7f25d00946c0 con 0x7f25d80fd770 2023-06-23T01:07:51.105 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-06-23T01:07:51.106 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":80,"fsid":"458642ee-1161-11ee-9b2b-001a4aab830c","created":"2023-06-23T01:00:57.877538+0000","modified":"2023-06-23T01:07:50.368379+0000","last_up_change":"2023-06-23T01:07:07.914869+0000","last_in_change":"2023-06-23T01:06:51.140023+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:04:00.299796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-06-23T01:07:48.883010+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-06-23T01:07:49.722003+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f28767ce-a780-4e50-b118-37a8c275c0e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6803","nonce":709285010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6805","nonce":709285010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6809","nonce":709285010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":709285010},{"type":"v1","addr":"172.21.15.46:6807","nonce":709285010}]},"public_addr":"172.21.15.46:6803/709285010","cluster_addr":"172.21.15.46:6805/709285010","heartbeat_back_addr":"172.21.15.46:6809/709285010","heartbeat_front_addr":"172.21.15.46:6807/709285010","state":["exists","up"]},{"osd":1,"uuid":"4d4c4c36-d4a1-4b96-9059-adffee04401a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6801","nonce":4266072183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6803","nonce":4266072183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6807","nonce":4266072183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":4266072183},{"type":"v1","addr":"172.21.15.92:6805","nonce":4266072183}]},"public_addr":"172.21.15.92:6801/4266072183","cluster_addr":"172.21.15.92:6803/4266072183","heartbeat_back_addr":"172.21.15.92:6807/4266072183","heartbeat_front_addr":"172.21.15.92:6805/4266072183","state":["exists","up"]},{"osd":2,"uuid":"ec7b0c60-88ea-4c2b-a34c-96d188e02014","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6801","nonce":1787957441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6803","nonce":1787957441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6807","nonce":1787957441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":1787957441},{"type":"v1","addr":"172.21.15.131:6805","nonce":1787957441}]},"public_addr":"172.21.15.131:6801/1787957441","cluster_addr":"172.21.15.131:6803/1787957441","heartbeat_back_addr":"172.21.15.131:6807/1787957441","heartbeat_front_addr":"172.21.15.131:6805/1787957441","state":["exists","up"]},{"osd":3,"uuid":"32670a23-3286-4538-9628-38d763429cbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6811","nonce":338690444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6813","nonce":338690444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6817","nonce":338690444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":338690444},{"type":"v1","addr":"172.21.15.46:6815","nonce":338690444}]},"public_addr":"172.21.15.46:6811/338690444","cluster_addr":"172.21.15.46:6813/338690444","heartbeat_back_addr":"172.21.15.46:6817/338690444","heartbeat_front_addr":"172.21.15.46:6815/338690444","state":["exists","up"]},{"osd":4,"uuid":"0d8f982c-eb74-48a2-949e-d0ccd7732937","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6809","nonce":2024629902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6811","nonce":2024629902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6815","nonce":2024629902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2024629902},{"type":"v1","addr":"172.21.15.92:6813","nonce":2024629902}]},"public_addr":"172.21.15.92:6809/2024629902","cluster_addr":"172.21.15.92:6811/2024629902","heartbeat_back_addr":"172.21.15.92:6815/2024629902","heartbeat_front_addr":"172.21.15.92:6813/2024629902","state":["exists","up"]},{"osd":5,"uuid":"52a7160c-4735-4a77-bc08-4d411c282b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6809","nonce":2577961971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6811","nonce":2577961971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6815","nonce":2577961971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":2577961971},{"type":"v1","addr":"172.21.15.131:6813","nonce":2577961971}]},"public_addr":"172.21.15.131:6809/2577961971","cluster_addr":"172.21.15.131:6811/2577961971","heartbeat_back_addr":"172.21.15.131:6815/2577961971","heartbeat_front_addr":"172.21.15.131:6813/2577961971","state":["exists","up"]},{"osd":6,"uuid":"3f3d2690-f655-4dda-a36c-bc97f9d08a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6819","nonce":558288790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6821","nonce":558288790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6825","nonce":558288790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":558288790},{"type":"v1","addr":"172.21.15.46:6823","nonce":558288790}]},"public_addr":"172.21.15.46:6819/558288790","cluster_addr":"172.21.15.46:6821/558288790","heartbeat_back_addr":"172.21.15.46:6825/558288790","heartbeat_front_addr":"172.21.15.46:6823/558288790","state":["exists","up"]},{"osd":7,"uuid":"2eb43609-8921-4891-b0c4-af932bfb69bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6817","nonce":3796540277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6819","nonce":3796540277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6823","nonce":3796540277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":3796540277},{"type":"v1","addr":"172.21.15.92:6821","nonce":3796540277}]},"public_addr":"172.21.15.92:6817/3796540277","cluster_addr":"172.21.15.92:6819/3796540277","heartbeat_back_addr":"172.21.15.92:6823/3796540277","heartbeat_front_addr":"172.21.15.92:6821/3796540277","state":["exists","up"]},{"osd":8,"uuid":"0139b39e-f933-4685-b056-a6c27fa87fd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6817","nonce":1004740496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6819","nonce":1004740496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6823","nonce":1004740496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":1004740496},{"type":"v1","addr":"172.21.15.131:6821","nonce":1004740496}]},"public_addr":"172.21.15.131:6817/1004740496","cluster_addr":"172.21.15.131:6819/1004740496","heartbeat_back_addr":"172.21.15.131:6823/1004740496","heartbeat_front_addr":"172.21.15.131:6821/1004740496","state":["exists","up"]},{"osd":9,"uuid":"8bd083b4-f766-4e1b-b959-19d2fa82d77f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6827","nonce":3042170676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6829","nonce":3042170676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6833","nonce":3042170676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3042170676},{"type":"v1","addr":"172.21.15.46:6831","nonce":3042170676}]},"public_addr":"172.21.15.46:6827/3042170676","cluster_addr":"172.21.15.46:6829/3042170676","heartbeat_back_addr":"172.21.15.46:6833/3042170676","heartbeat_front_addr":"172.21.15.46:6831/3042170676","state":["exists","up"]},{"osd":10,"uuid":"0b4f58f5-d971-4634-a1b6-896501d3d218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6825","nonce":2375910823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6827","nonce":2375910823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6831","nonce":2375910823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":2375910823},{"type":"v1","addr":"172.21.15.92:6829","nonce":2375910823}]},"public_addr":"172.21.15.92:6825/2375910823","cluster_addr":"172.21.15.92:6827/2375910823","heartbeat_back_addr":"172.21.15.92:6831/2375910823","heartbeat_front_addr":"172.21.15.92:6829/2375910823","state":["exists","up"]},{"osd":11,"uuid":"7d7a4115-c6e9-4cef-865b-05f57fbcface","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6825","nonce":1716414800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6826","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6827","nonce":1716414800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6830","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6831","nonce":1716414800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6828","nonce":1716414800},{"type":"v1","addr":"172.21.15.131:6829","nonce":1716414800}]},"public_addr":"172.21.15.131:6825/1716414800","cluster_addr":"172.21.15.131:6827/1716414800","heartbeat_back_addr":"172.21.15.131:6831/1716414800","heartbeat_front_addr":"172.21.15.131:6829/1716414800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:15.785940+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:36.479807+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:03:57.678238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:04:20.396101+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:04:41.179881+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:01.570524+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:22.313074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:05:43.129445+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:03.897470+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:25.196569+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:06:46.208577+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:07:06.788656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/685567953":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6801/2331157358":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6800/2331157358":"2023-06-24T01:01:55.786463+0000","172.21.15.46:0/1655911006":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/187730254":"2023-06-24T01:01:32.095580+0000","172.21.15.46:6801/3291477171":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/3127678959":"2023-06-24T01:01:55.786463+0000","172.21.15.46:6800/3291477171":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/411746526":"2023-06-24T01:01:32.095580+0000","172.21.15.46:0/360533287":"2023-06-24T01:01:55.786463+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:07:51.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.104+0000 7f25dfa82700 1 -- 172.21.15.46:0/3755172317 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f25c409b680 msgr2=0x7f25c409db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.104+0000 7f25dfa82700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f25c409b680 0x7f25c409db30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009b80 tx=0x7f25d4009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.104+0000 7f25dfa82700 1 -- 172.21.15.46:0/3755172317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d80fd770 msgr2=0x7f25d8114c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.104+0000 7f25dfa82700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d80fd770 0x7f25d8114c70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f25d000ec30 tx=0x7f25d000b040 comp rx=0 tx=0).stop 2023-06-23T01:07:51.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.105+0000 7f25dfa82700 1 -- 172.21.15.46:0/3755172317 shutdown_connections 2023-06-23T01:07:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.105+0000 7f25dfa82700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25d80fd770 0x7f25d8114c70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.105+0000 7f25dfa82700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f25d80fbbd0 0x7f25d8065a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.105+0000 7f25dfa82700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f25c409b680 0x7f25c409db30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.105+0000 7f25dfa82700 1 --2- 172.21.15.46:0/3755172317 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25d80fcde0 0x7f25d8112630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.105+0000 7f25dfa82700 1 -- 172.21.15.46:0/3755172317 >> 172.21.15.46:0/3755172317 conn(0x7f25d80f7310 msgr2=0x7f25d80f95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.105+0000 7f25dfa82700 1 -- 172.21.15.46:0/3755172317 shutdown_connections 2023-06-23T01:07:51.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.105+0000 7f25dfa82700 1 -- 172.21.15.46:0/3755172317 wait complete. 2023-06-23T01:07:51.120 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-23T01:07:51.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:50 smithi131 ceph-mon[86495]: pgmap v245: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:51.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:50 smithi131 ceph-mon[86495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-06-23T01:07:51.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:50 smithi131 ceph-mon[86495]: osdmap e79: 12 total, 12 up, 12 in 2023-06-23T01:07:51.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:50 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/3505635919' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-06-23T01:07:51.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:50 smithi131 ceph-mon[86495]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-06-23T01:07:51.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:50 smithi131 ceph-mon[86495]: osdmap e80: 12 total, 12 up, 12 in 2023-06-23T01:07:51.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:50 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/3505635919' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-06-23T01:07:51.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:50 smithi131 ceph-mon[86495]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-06-23T01:07:51.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:50 smithi131 ceph-mon[86495]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-06-23T01:07:51.233 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:50 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/191662845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:51.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.249+0000 7fb24e84f700 1 -- 172.21.15.46:0/3538425179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2480fe870 msgr2=0x7fb248105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:51.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.249+0000 7fb24e84f700 1 --2- 172.21.15.46:0/3538425179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2480fe870 0x7fb248105130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb240009d60 tx=0x7fb240004d10 comp rx=0 tx=0).stop 2023-06-23T01:07:51.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 -- 172.21.15.46:0/3538425179 shutdown_connections 2023-06-23T01:07:51.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 --2- 172.21.15.46:0/3538425179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2480fe870 0x7fb248105130 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 --2- 172.21.15.46:0/3538425179 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb2480fccd0 0x7fb2480fd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 --2- 172.21.15.46:0/3538425179 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2480fdee0 0x7fb2480fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 -- 172.21.15.46:0/3538425179 >> 172.21.15.46:0/3538425179 conn(0x7fb2480f8450 msgr2=0x7fb2480fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:51.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 -- 172.21.15.46:0/3538425179 shutdown_connections 2023-06-23T01:07:51.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 -- 172.21.15.46:0/3538425179 wait complete. 2023-06-23T01:07:51.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 Processor -- start 2023-06-23T01:07:51.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 -- start start 2023-06-23T01:07:51.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2480fccd0 0x7fb248071720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:51.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2480fdee0 0x7fb24806fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:51.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb2480fe870 0x7fb2480702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:51.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb248070a30 con 0x7fb2480fccd0 2023-06-23T01:07:51.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb248070ba0 con 0x7fb2480fdee0 2023-06-23T01:07:51.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.250+0000 7fb24e84f700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2481176e0 con 0x7fb2480fe870 2023-06-23T01:07:51.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb247fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2480fccd0 0x7fb248071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb247fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2480fccd0 0x7fb248071720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60566/0 (socket says 172.21.15.46:60566) 2023-06-23T01:07:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb247fff700 1 -- 172.21.15.46:0/2377765383 learned_addr learned my addr 172.21.15.46:0/2377765383 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb24cdec700 1 --2- 172.21.15.46:0/2377765383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb2480fe870 0x7fb2480702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb2477fe700 1 --2- 172.21.15.46:0/2377765383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2480fdee0 0x7fb24806fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb247fff700 1 -- 172.21.15.46:0/2377765383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb2480fe870 msgr2=0x7fb2480702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb247fff700 1 --2- 172.21.15.46:0/2377765383 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7fb2480fe870 0x7fb2480702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb247fff700 1 -- 172.21.15.46:0/2377765383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2480fdee0 msgr2=0x7fb24806fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb247fff700 1 --2- 172.21.15.46:0/2377765383 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb2480fdee0 0x7fb24806fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb247fff700 1 -- 172.21.15.46:0/2377765383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb240009940 con 0x7fb2480fccd0 2023-06-23T01:07:51.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb247fff700 1 --2- 172.21.15.46:0/2377765383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2480fccd0 0x7fb248071720 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb23800ebf0 tx=0x7fb23800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:51.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb244ff9700 1 -- 172.21.15.46:0/2377765383 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2380047d0 con 0x7fb2480fccd0 2023-06-23T01:07:51.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb244ff9700 1 -- 172.21.15.46:0/2377765383 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb23800f040 con 0x7fb2480fccd0 2023-06-23T01:07:51.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb24e84f700 1 -- 172.21.15.46:0/2377765383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb248117930 con 0x7fb2480fccd0 2023-06-23T01:07:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb244ff9700 1 -- 172.21.15.46:0/2377765383 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23800c820 con 0x7fb2480fccd0 2023-06-23T01:07:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.251+0000 7fb24e84f700 1 -- 172.21.15.46:0/2377765383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb248117ef0 con 0x7fb2480fccd0 2023-06-23T01:07:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.252+0000 7fb24e84f700 1 -- 172.21.15.46:0/2377765383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb248103950 con 0x7fb2480fccd0 2023-06-23T01:07:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.253+0000 7fb244ff9700 1 -- 172.21.15.46:0/2377765383 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7fb23800cc50 con 0x7fb2480fccd0 2023-06-23T01:07:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.254+0000 7fb244ff9700 1 --2- 172.21.15.46:0/2377765383 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb23009b7b0 0x7fb23009dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.254+0000 7fb2477fe700 1 --2- 172.21.15.46:0/2377765383 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb23009b7b0 0x7fb23009dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.254+0000 7fb244ff9700 1 -- 172.21.15.46:0/2377765383 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7975+0+0 (secure 0 0 0) 0x7fb2380c47a0 con 0x7fb2480fccd0 2023-06-23T01:07:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.254+0000 7fb2477fe700 1 --2- 172.21.15.46:0/2377765383 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7fb23009b7b0 0x7fb23009dc60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009b80 tx=0x7fb23c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.257+0000 7fb244ff9700 1 -- 172.21.15.46:0/2377765383 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7fb23804da00 con 0x7fb2480fccd0 2023-06-23T01:07:51.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:50 smithi092 ceph-mon[85670]: pgmap v245: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-06-23T01:07:51.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:50 smithi092 ceph-mon[85670]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-06-23T01:07:51.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:50 smithi092 ceph-mon[85670]: osdmap e79: 12 total, 12 up, 12 in 2023-06-23T01:07:51.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:50 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/3505635919' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-06-23T01:07:51.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:50 smithi092 ceph-mon[85670]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-06-23T01:07:51.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:50 smithi092 ceph-mon[85670]: osdmap e80: 12 total, 12 up, 12 in 2023-06-23T01:07:51.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:50 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/3505635919' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-06-23T01:07:51.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:50 smithi092 ceph-mon[85670]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-06-23T01:07:51.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:50 smithi092 ceph-mon[85670]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-06-23T01:07:51.391 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:50 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/191662845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:51.529 INFO:teuthology.orchestra.run.smithi046.stderr:no valid command found; 1 closest matches: 2023-06-23T01:07:51.530 INFO:teuthology.orchestra.run.smithi046.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T01:07:51.530 INFO:teuthology.orchestra.run.smithi046.stderr:Error EINVAL: invalid command 2023-06-23T01:07:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T01:07:51.533 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300, 'standby_replay': True} 2023-06-23T01:07:51.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-06-23T01:07:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.665+0000 7f6e135fc700 1 -- 172.21.15.46:0/3160820971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e0c0fce30 msgr2=0x7f6e0c0fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.665+0000 7f6e135fc700 1 --2- 172.21.15.46:0/3160820971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e0c0fce30 0x7f6e0c0fd280 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0049c0 tx=0x7f6dfc009e30 comp rx=0 tx=0).stop 2023-06-23T01:07:51.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.665+0000 7f6e135fc700 1 -- 172.21.15.46:0/3160820971 shutdown_connections 2023-06-23T01:07:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.665+0000 7f6e135fc700 1 --2- 172.21.15.46:0/3160820971 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e0c0fd7c0 0x7f6e0c104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.665+0000 7f6e135fc700 1 --2- 172.21.15.46:0/3160820971 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e0c0fbc20 0x7f6e0c0fc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.665+0000 7f6e135fc700 1 --2- 172.21.15.46:0/3160820971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e0c0fce30 0x7f6e0c0fd280 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.665+0000 7f6e135fc700 1 -- 172.21.15.46:0/3160820971 >> 172.21.15.46:0/3160820971 conn(0x7f6e0c0f73a0 msgr2=0x7f6e0c0f97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:07:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.665+0000 7f6e135fc700 1 -- 172.21.15.46:0/3160820971 shutdown_connections 2023-06-23T01:07:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.665+0000 7f6e135fc700 1 -- 172.21.15.46:0/3160820971 wait complete. 2023-06-23T01:07:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e135fc700 1 Processor -- start 2023-06-23T01:07:51.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e135fc700 1 -- start start 2023-06-23T01:07:51.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e135fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e0c0fbc20 0x7f6e0c10d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:51.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e135fc700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e0c0fce30 0x7f6e0c10dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:51.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e135fc700 1 --2- >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e0c0fd7c0 0x7f6e0c114d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:51.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e135fc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c115320 con 0x7f6e0c0fbc20 2023-06-23T01:07:51.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e135fc700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c115490 con 0x7f6e0c0fce30 2023-06-23T01:07:51.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e135fc700 1 -- --> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c115650 con 0x7f6e0c0fd7c0 2023-06-23T01:07:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e11398700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e0c0fbc20 0x7f6e0c10d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e11398700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e0c0fbc20 0x7f6e0c10d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60574/0 (socket says 172.21.15.46:60574) 2023-06-23T01:07:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e11398700 1 -- 172.21.15.46:0/3172943210 learned_addr learned my addr 172.21.15.46:0/3172943210 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-06-23T01:07:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e10b97700 1 --2- 172.21.15.46:0/3172943210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e0c0fce30 0x7f6e0c10dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e11b99700 1 --2- 172.21.15.46:0/3172943210 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e0c0fd7c0 0x7f6e0c114d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e11398700 1 -- 172.21.15.46:0/3172943210 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e0c0fd7c0 msgr2=0x7f6e0c114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e11398700 1 --2- 172.21.15.46:0/3172943210 >> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] conn(0x7f6e0c0fd7c0 0x7f6e0c114d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e11398700 1 -- 172.21.15.46:0/3172943210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e0c0fce30 msgr2=0x7f6e0c10dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:07:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e11398700 1 --2- 172.21.15.46:0/3172943210 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f6e0c0fce30 0x7f6e0c10dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:07:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e11398700 1 -- 172.21.15.46:0/3172943210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc005040 con 0x7f6e0c0fbc20 2023-06-23T01:07:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.666+0000 7f6e11398700 1 --2- 172.21.15.46:0/3172943210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e0c0fbc20 0x7f6e0c10d740 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6e0800ebf0 tx=0x7f6e0800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:51.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.667+0000 7f6e01ffb700 1 -- 172.21.15.46:0/3172943210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e08007cc0 con 0x7f6e0c0fbc20 2023-06-23T01:07:51.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.667+0000 7f6e01ffb700 1 -- 172.21.15.46:0/3172943210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6e0800f040 con 0x7f6e0c0fbc20 2023-06-23T01:07:51.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.667+0000 7f6e135fc700 1 -- 172.21.15.46:0/3172943210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0c1158d0 con 0x7f6e0c0fbc20 2023-06-23T01:07:51.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.667+0000 7f6e01ffb700 1 -- 172.21.15.46:0/3172943210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0800c800 con 0x7f6e0c0fbc20 2023-06-23T01:07:51.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.667+0000 7f6e135fc700 1 -- 172.21.15.46:0/3172943210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0c115e90 con 0x7f6e0c0fbc20 2023-06-23T01:07:51.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.668+0000 7f6e135fc700 1 -- 172.21.15.46:0/3172943210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c102fb0 con 0x7f6e0c0fbc20 2023-06-23T01:07:51.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.669+0000 7f6e01ffb700 1 -- 172.21.15.46:0/3172943210 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 129201+0+0 (secure 0 0 0) 0x7f6e0800cc30 con 0x7f6e0c0fbc20 2023-06-23T01:07:51.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.669+0000 7f6e01ffb700 1 --2- 172.21.15.46:0/3172943210 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6df809b7b0 0x7f6df809dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:07:51.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.669+0000 7f6e10b97700 1 --2- 172.21.15.46:0/3172943210 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6df809b7b0 0x7f6df809dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:07:51.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.670+0000 7f6e01ffb700 1 -- 172.21.15.46:0/3172943210 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f6e080c4630 con 0x7f6e0c0fbc20 2023-06-23T01:07:51.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.670+0000 7f6e10b97700 1 --2- 172.21.15.46:0/3172943210 >> [v2:172.21.15.46:6800/1578446983,v1:172.21.15.46:6801/1578446983] conn(0x7f6df809b7b0 0x7f6df809dc60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc007f50 tx=0x7f6dfc009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:07:51.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-06-23T01:07:51.674+0000 7f6e01ffb700 1 -- 172.21.15.46:0/3172943210 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174561 (secure 0 0 0) 0x7f6e0804da00 con 0x7f6e0c0fbc20 2023-06-23T01:07:51.928 INFO:teuthology.orchestra.run.smithi046.stderr:no valid command found; 1 closest matches: 2023-06-23T01:07:51.928 INFO:teuthology.orchestra.run.smithi046.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T01:07:51.928 INFO:teuthology.orchestra.run.smithi046.stderr:Error EINVAL: invalid command 2023-06-23T01:07:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T01:07:51.930 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 896, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true' 2023-06-23T01:07:51.931 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-23T01:07:51.959 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-23T01:07:51.988 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-23T01:07:52.018 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-06-23T01:07:52.019 DEBUG:teuthology.orchestra.run.smithi046:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-06-23T01:07:52.034 DEBUG:teuthology.orchestra.run.smithi092:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-06-23T01:07:52.049 DEBUG:teuthology.orchestra.run.smithi131:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-06-23T01:07:52.073 INFO:tasks.cephadm:Stopping all daemons... 2023-06-23T01:07:52.073 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-06-23T01:07:52.074 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.a 2023-06-23T01:07:52.231 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:51 smithi131 ceph-mon[86495]: from='client.? 172.21.15.46:0/3755172317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:52.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:51 smithi131 ceph-mon[86495]: mds.? [v2:172.21.15.92:6838/3591784213,v1:172.21.15.92:6839/3591784213] up:active 2023-06-23T01:07:52.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:51 smithi131 ceph-mon[86495]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-06-23T01:07:52.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:51 smithi131 ceph-mon[86495]: osdmap e81: 12 total, 12 up, 12 in 2023-06-23T01:07:52.232 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:51 smithi131 ceph-mon[86495]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-06-23T01:07:52.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:51 smithi046 ceph-mon[84949]: from='client.? 172.21.15.46:0/3755172317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:52.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:51 smithi046 ceph-mon[84949]: mds.? [v2:172.21.15.92:6838/3591784213,v1:172.21.15.92:6839/3591784213] up:active 2023-06-23T01:07:52.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:51 smithi046 ceph-mon[84949]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-06-23T01:07:52.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:51 smithi046 ceph-mon[84949]: osdmap e81: 12 total, 12 up, 12 in 2023-06-23T01:07:52.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:51 smithi046 ceph-mon[84949]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-06-23T01:07:52.319 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:52 smithi046 systemd[1]: Stopping Ceph mon.a for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:52.389 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:51 smithi092 ceph-mon[85670]: from='client.? 172.21.15.46:0/3755172317' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-06-23T01:07:52.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:51 smithi092 ceph-mon[85670]: mds.? [v2:172.21.15.92:6838/3591784213,v1:172.21.15.92:6839/3591784213] up:active 2023-06-23T01:07:52.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:51 smithi092 ceph-mon[85670]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-06-23T01:07:52.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:51 smithi092 ceph-mon[85670]: osdmap e81: 12 total, 12 up, 12 in 2023-06-23T01:07:52.390 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:51 smithi092 ceph-mon[85670]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-06-23T01:07:52.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:52 smithi046 conmon[84925]: 2023-06-23T01:07:52.433+0000 7fdd7ccd6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:07:52.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Jun 23 01:07:52 smithi046 conmon[84925]: 2023-06-23T01:07:52.433+0000 7fdd7ccd6700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-06-23T01:07:52.929 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.a.service' 2023-06-23T01:07:52.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:07:52.982 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-06-23T01:07:52.982 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-06-23T01:07:52.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.b 2023-06-23T01:07:53.324 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:53 smithi092 systemd[1]: Stopping Ceph mon.b for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:53.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:53 smithi092 conmon[85646]: 2023-06-23T01:07:53.319+0000 7f58de225700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:07:53.640 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:53 smithi092 conmon[85646]: 2023-06-23T01:07:53.319+0000 7f58de225700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-06-23T01:07:53.919 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.b.service' 2023-06-23T01:07:53.944 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:53 smithi092 bash[119963]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mon-b 2023-06-23T01:07:53.945 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:53 smithi092 systemd[1]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.b.service: Succeeded. 2023-06-23T01:07:53.945 INFO:journalctl@ceph.mon.b.smithi092.stdout:Jun 23 01:07:53 smithi092 systemd[1]: Stopped Ceph mon.b for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:07:53.973 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:07:53.974 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-06-23T01:07:53.974 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-06-23T01:07:53.974 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.c 2023-06-23T01:07:54.321 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:54 smithi131 systemd[1]: Stopping Ceph mon.c for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:54.652 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:54 smithi131 conmon[86430]: 2023-06-23T01:07:54.316+0000 7f1137b83700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:07:54.653 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:54 smithi131 conmon[86430]: 2023-06-23T01:07:54.316+0000 7f1137b83700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-06-23T01:07:54.982 INFO:journalctl@ceph.mon.c.smithi131.stdout:Jun 23 01:07:54 smithi131 bash[122387]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mon-c 2023-06-23T01:07:55.023 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mon.c.service' 2023-06-23T01:07:55.072 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:07:55.072 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-06-23T01:07:55.073 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-06-23T01:07:55.073 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.x 2023-06-23T01:07:55.435 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:07:55 smithi046 systemd[1]: Stopping Ceph mgr.x for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:55.719 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:07:55 smithi046 conmon[85362]: 2023-06-23T01:07:55.432+0000 7fb2c446d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:07:55.719 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Jun 23 01:07:55 smithi046 conmon[85362]: 2023-06-23T01:07:55.432+0000 7fb2c446d700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-06-23T01:07:55.933 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.x.service' 2023-06-23T01:07:55.971 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:07:55.973 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-06-23T01:07:55.974 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-06-23T01:07:55.974 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.y 2023-06-23T01:07:56.389 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:07:56 smithi092 systemd[1]: Stopping Ceph mgr.y for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:56.806 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:07:56 smithi092 bash[120249]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mgr-y 2023-06-23T01:07:56.806 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Jun 23 01:07:56 smithi092 systemd[1]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-06-23T01:07:56.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.y.service' 2023-06-23T01:07:56.848 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:07:56.849 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-06-23T01:07:56.850 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-06-23T01:07:56.850 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.z 2023-06-23T01:07:57.232 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:07:56 smithi131 systemd[1]: Stopping Ceph mgr.z for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:57.694 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:07:57 smithi131 bash[122673]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mgr-z 2023-06-23T01:07:57.694 INFO:journalctl@ceph.mgr.z.smithi131.stdout:Jun 23 01:07:57 smithi131 systemd[1]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-06-23T01:07:57.700 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mgr.z.service' 2023-06-23T01:07:57.737 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:07:57.737 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-06-23T01:07:57.738 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-06-23T01:07:57.738 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.0 2023-06-23T01:07:58.068 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:07:57 smithi046 systemd[1]: Stopping Ceph osd.0 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:07:58.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:07:58 smithi046 conmon[101888]: 2023-06-23T01:07:58.090+0000 7fa5cad8a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:07:58.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:07:58 smithi046 conmon[101888]: 2023-06-23T01:07:58.090+0000 7fa5cad8a700 -1 osd.0 81 *** Got signal Terminated *** 2023-06-23T01:07:58.568 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:07:58 smithi046 conmon[101888]: 2023-06-23T01:07:58.090+0000 7fa5cad8a700 -1 osd.0 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:08:04.068 INFO:journalctl@ceph.osd.0.smithi046.stdout:Jun 23 01:08:03 smithi046 bash[133314]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-0 2023-06-23T01:08:05.168 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.0.service' 2023-06-23T01:08:05.208 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:08:05.208 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-06-23T01:08:05.209 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-06-23T01:08:05.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.1 2023-06-23T01:08:05.629 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:08:05 smithi092 systemd[1]: Stopping Ceph osd.1 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:08:05.890 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:08:05 smithi092 conmon[92152]: 2023-06-23T01:08:05.626+0000 7f5bd6a87700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:08:05.890 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:08:05 smithi092 conmon[92152]: 2023-06-23T01:08:05.626+0000 7f5bd6a87700 -1 osd.1 81 *** Got signal Terminated *** 2023-06-23T01:08:05.890 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:08:05 smithi092 conmon[92152]: 2023-06-23T01:08:05.626+0000 7f5bd6a87700 -1 osd.1 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:08:11.389 INFO:journalctl@ceph.osd.1.smithi092.stdout:Jun 23 01:08:11 smithi092 bash[120524]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-1 2023-06-23T01:08:12.557 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.1.service' 2023-06-23T01:08:12.596 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:08:12.598 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-06-23T01:08:12.598 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-06-23T01:08:12.599 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.2 2023-06-23T01:08:12.942 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:08:12 smithi131 systemd[1]: Stopping Ceph osd.2 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:08:13.232 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:08:12 smithi131 conmon[94162]: 2023-06-23T01:08:12.938+0000 7f2bc1e2b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:08:13.232 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:08:12 smithi131 conmon[94162]: 2023-06-23T01:08:12.938+0000 7f2bc1e2b700 -1 osd.2 81 *** Got signal Terminated *** 2023-06-23T01:08:13.232 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:08:12 smithi131 conmon[94162]: 2023-06-23T01:08:12.938+0000 7f2bc1e2b700 -1 osd.2 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:08:18.732 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:08:18 smithi131 bash[122963]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-2 2023-06-23T01:08:20.166 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.2.service' 2023-06-23T01:08:20.190 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:08:20 smithi131 systemd[1]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.2.service: Succeeded. 2023-06-23T01:08:20.190 INFO:journalctl@ceph.osd.2.smithi131.stdout:Jun 23 01:08:20 smithi131 systemd[1]: Stopped Ceph osd.2 for 458642ee-1161-11ee-9b2b-001a4aab830c. 2023-06-23T01:08:20.204 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:08:20.205 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-06-23T01:08:20.205 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-06-23T01:08:20.206 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.3 2023-06-23T01:08:20.559 INFO:journalctl@ceph.osd.3.smithi046.stdout:Jun 23 01:08:20 smithi046 systemd[1]: Stopping Ceph osd.3 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:08:20.818 INFO:journalctl@ceph.osd.3.smithi046.stdout:Jun 23 01:08:20 smithi046 conmon[108845]: 2023-06-23T01:08:20.554+0000 7f14382c8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:08:20.818 INFO:journalctl@ceph.osd.3.smithi046.stdout:Jun 23 01:08:20 smithi046 conmon[108845]: 2023-06-23T01:08:20.554+0000 7f14382c8700 -1 osd.3 81 *** Got signal Terminated *** 2023-06-23T01:08:20.818 INFO:journalctl@ceph.osd.3.smithi046.stdout:Jun 23 01:08:20 smithi046 conmon[108845]: 2023-06-23T01:08:20.554+0000 7f14382c8700 -1 osd.3 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:08:26.568 INFO:journalctl@ceph.osd.3.smithi046.stdout:Jun 23 01:08:26 smithi046 bash[133871]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-3 2023-06-23T01:08:27.786 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.3.service' 2023-06-23T01:08:27.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:08:27.826 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-06-23T01:08:27.826 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-06-23T01:08:27.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.4 2023-06-23T01:08:28.139 INFO:journalctl@ceph.osd.4.smithi092.stdout:Jun 23 01:08:27 smithi092 systemd[1]: Stopping Ceph osd.4 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:08:28.639 INFO:journalctl@ceph.osd.4.smithi092.stdout:Jun 23 01:08:28 smithi092 conmon[99058]: 2023-06-23T01:08:28.187+0000 7f8018a6f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:08:28.640 INFO:journalctl@ceph.osd.4.smithi092.stdout:Jun 23 01:08:28 smithi092 conmon[99058]: 2023-06-23T01:08:28.187+0000 7f8018a6f700 -1 osd.4 81 *** Got signal Terminated *** 2023-06-23T01:08:28.640 INFO:journalctl@ceph.osd.4.smithi092.stdout:Jun 23 01:08:28 smithi092 conmon[99058]: 2023-06-23T01:08:28.187+0000 7f8018a6f700 -1 osd.4 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:08:34.139 INFO:journalctl@ceph.osd.4.smithi092.stdout:Jun 23 01:08:33 smithi092 bash[121077]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-4 2023-06-23T01:08:35.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.4.service' 2023-06-23T01:08:35.416 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:08:35.417 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-06-23T01:08:35.417 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-06-23T01:08:35.417 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.5 2023-06-23T01:08:35.732 INFO:journalctl@ceph.osd.5.smithi131.stdout:Jun 23 01:08:35 smithi131 systemd[1]: Stopping Ceph osd.5 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:08:36.232 INFO:journalctl@ceph.osd.5.smithi131.stdout:Jun 23 01:08:35 smithi131 conmon[101153]: 2023-06-23T01:08:35.760+0000 7f1bde14f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:08:36.232 INFO:journalctl@ceph.osd.5.smithi131.stdout:Jun 23 01:08:35 smithi131 conmon[101153]: 2023-06-23T01:08:35.760+0000 7f1bde14f700 -1 osd.5 81 *** Got signal Terminated *** 2023-06-23T01:08:36.232 INFO:journalctl@ceph.osd.5.smithi131.stdout:Jun 23 01:08:35 smithi131 conmon[101153]: 2023-06-23T01:08:35.760+0000 7f1bde14f700 -1 osd.5 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:08:41.481 INFO:journalctl@ceph.osd.5.smithi131.stdout:Jun 23 01:08:41 smithi131 bash[123508]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-5 2023-06-23T01:08:42.778 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.5.service' 2023-06-23T01:08:42.818 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:08:42.819 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-06-23T01:08:42.819 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-06-23T01:08:42.820 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.6 2023-06-23T01:08:43.196 INFO:journalctl@ceph.osd.6.smithi046.stdout:Jun 23 01:08:42 smithi046 systemd[1]: Stopping Ceph osd.6 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:08:43.568 INFO:journalctl@ceph.osd.6.smithi046.stdout:Jun 23 01:08:43 smithi046 conmon[116055]: 2023-06-23T01:08:43.192+0000 7f18aa20f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:08:43.568 INFO:journalctl@ceph.osd.6.smithi046.stdout:Jun 23 01:08:43 smithi046 conmon[116055]: 2023-06-23T01:08:43.192+0000 7f18aa20f700 -1 osd.6 81 *** Got signal Terminated *** 2023-06-23T01:08:43.568 INFO:journalctl@ceph.osd.6.smithi046.stdout:Jun 23 01:08:43 smithi046 conmon[116055]: 2023-06-23T01:08:43.192+0000 7f18aa20f700 -1 osd.6 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:08:49.068 INFO:journalctl@ceph.osd.6.smithi046.stdout:Jun 23 01:08:48 smithi046 bash[134427]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-6 2023-06-23T01:08:50.197 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.6.service' 2023-06-23T01:08:50.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:08:50.240 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-06-23T01:08:50.241 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-06-23T01:08:50.241 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.7 2023-06-23T01:08:50.594 INFO:journalctl@ceph.osd.7.smithi092.stdout:Jun 23 01:08:50 smithi092 systemd[1]: Stopping Ceph osd.7 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:08:50.889 INFO:journalctl@ceph.osd.7.smithi092.stdout:Jun 23 01:08:50 smithi092 conmon[106130]: 2023-06-23T01:08:50.590+0000 7f5760a80700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:08:50.890 INFO:journalctl@ceph.osd.7.smithi092.stdout:Jun 23 01:08:50 smithi092 conmon[106130]: 2023-06-23T01:08:50.590+0000 7f5760a80700 -1 osd.7 81 *** Got signal Terminated *** 2023-06-23T01:08:50.890 INFO:journalctl@ceph.osd.7.smithi092.stdout:Jun 23 01:08:50 smithi092 conmon[106130]: 2023-06-23T01:08:50.590+0000 7f5760a80700 -1 osd.7 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:08:56.389 INFO:journalctl@ceph.osd.7.smithi092.stdout:Jun 23 01:08:56 smithi092 bash[121618]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-7 2023-06-23T01:08:57.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.7.service' 2023-06-23T01:08:57.633 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:08:57.634 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-06-23T01:08:57.634 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-06-23T01:08:57.635 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.8 2023-06-23T01:08:57.982 INFO:journalctl@ceph.osd.8.smithi131.stdout:Jun 23 01:08:57 smithi131 systemd[1]: Stopping Ceph osd.8 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:08:58.481 INFO:journalctl@ceph.osd.8.smithi131.stdout:Jun 23 01:08:57 smithi131 conmon[108138]: 2023-06-23T01:08:57.981+0000 7f43991b0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:08:58.482 INFO:journalctl@ceph.osd.8.smithi131.stdout:Jun 23 01:08:57 smithi131 conmon[108138]: 2023-06-23T01:08:57.981+0000 7f43991b0700 -1 osd.8 81 *** Got signal Terminated *** 2023-06-23T01:08:58.482 INFO:journalctl@ceph.osd.8.smithi131.stdout:Jun 23 01:08:57 smithi131 conmon[108138]: 2023-06-23T01:08:57.981+0000 7f43991b0700 -1 osd.8 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:09:03.982 INFO:journalctl@ceph.osd.8.smithi131.stdout:Jun 23 01:09:03 smithi131 bash[124044]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-8 2023-06-23T01:09:05.089 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.8.service' 2023-06-23T01:09:05.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:09:05.129 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-06-23T01:09:05.129 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-06-23T01:09:05.130 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.9 2023-06-23T01:09:05.485 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:09:05 smithi046 systemd[1]: Stopping Ceph osd.9 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:09:05.485 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:09:05 smithi046 conmon[123172]: 2023-06-23T01:09:05.335+0000 7f60a2863700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:02.734563+0000 front 2023-06-23T01:08:02.734586+0000 (oldest deadline 2023-06-23T01:09:04.434693+0000) 2023-06-23T01:09:05.485 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:09:05 smithi046 conmon[123172]: 2023-06-23T01:09:05.482+0000 7f60a5263700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:09:05.818 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:09:05 smithi046 conmon[123172]: 2023-06-23T01:09:05.482+0000 7f60a5263700 -1 osd.9 81 *** Got signal Terminated *** 2023-06-23T01:09:05.818 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:09:05 smithi046 conmon[123172]: 2023-06-23T01:09:05.482+0000 7f60a5263700 -1 osd.9 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:09:06.568 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:09:06 smithi046 conmon[123172]: 2023-06-23T01:09:06.297+0000 7f60a2863700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:02.734563+0000 front 2023-06-23T01:08:02.734586+0000 (oldest deadline 2023-06-23T01:09:04.434693+0000) 2023-06-23T01:09:06.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:06 smithi092 conmon[113172]: 2023-06-23T01:09:06.313+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:07.568 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:09:07 smithi046 conmon[123172]: 2023-06-23T01:09:07.292+0000 7f60a2863700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:02.734563+0000 front 2023-06-23T01:08:02.734586+0000 (oldest deadline 2023-06-23T01:09:04.434693+0000) 2023-06-23T01:09:07.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:07 smithi092 conmon[113172]: 2023-06-23T01:09:07.322+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:08.481 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:08 smithi131 conmon[115245]: 2023-06-23T01:09:08.216+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:08.568 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:09:08 smithi046 conmon[123172]: 2023-06-23T01:09:08.275+0000 7f60a2863700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:02.734563+0000 front 2023-06-23T01:08:02.734586+0000 (oldest deadline 2023-06-23T01:09:04.434693+0000) 2023-06-23T01:09:08.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:08 smithi092 conmon[113172]: 2023-06-23T01:09:08.341+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:09.481 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:09 smithi131 conmon[115245]: 2023-06-23T01:09:09.188+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:09.568 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:09:09 smithi046 conmon[123172]: 2023-06-23T01:09:09.262+0000 7f60a2863700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:02.734563+0000 front 2023-06-23T01:08:02.734586+0000 (oldest deadline 2023-06-23T01:09:04.434693+0000) 2023-06-23T01:09:09.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:09 smithi092 conmon[113172]: 2023-06-23T01:09:09.311+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:10.481 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:10 smithi131 conmon[115245]: 2023-06-23T01:09:10.178+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:10.568 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:09:10 smithi046 conmon[123172]: 2023-06-23T01:09:10.246+0000 7f60a2863700 -1 osd.9 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:02.734563+0000 front 2023-06-23T01:08:02.734586+0000 (oldest deadline 2023-06-23T01:09:04.434693+0000) 2023-06-23T01:09:10.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:10 smithi092 conmon[113172]: 2023-06-23T01:09:10.294+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:11.481 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:11 smithi131 conmon[115245]: 2023-06-23T01:09:11.195+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:11.568 INFO:journalctl@ceph.osd.9.smithi046.stdout:Jun 23 01:09:11 smithi046 bash[134969]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-9 2023-06-23T01:09:11.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:11 smithi092 conmon[113172]: 2023-06-23T01:09:11.267+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:12.481 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:12 smithi131 conmon[115245]: 2023-06-23T01:09:12.219+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:12.604 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.9.service' 2023-06-23T01:09:12.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:12 smithi092 conmon[113172]: 2023-06-23T01:09:12.251+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:12.644 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:09:12.644 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-06-23T01:09:12.645 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-06-23T01:09:12.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.10 2023-06-23T01:09:13.006 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:12 smithi092 systemd[1]: Stopping Ceph osd.10 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:09:13.389 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:13 smithi092 conmon[113172]: 2023-06-23T01:09:13.003+0000 7fea72769700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:09:13.390 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:13 smithi092 conmon[113172]: 2023-06-23T01:09:13.003+0000 7fea72769700 -1 osd.10 81 *** Got signal Terminated *** 2023-06-23T01:09:13.390 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:13 smithi092 conmon[113172]: 2023-06-23T01:09:13.003+0000 7fea72769700 -1 osd.10 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:09:13.390 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:13 smithi092 conmon[113172]: 2023-06-23T01:09:13.234+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:13.390 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:13 smithi092 conmon[113172]: 2023-06-23T01:09:13.234+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:06.530505+0000 front 2023-06-23T01:08:06.530526+0000 (oldest deadline 2023-06-23T01:09:12.430698+0000) 2023-06-23T01:09:13.481 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:13 smithi131 conmon[115245]: 2023-06-23T01:09:13.219+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:13.482 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:13 smithi131 conmon[115245]: 2023-06-23T01:09:13.219+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:14.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:14 smithi092 conmon[113172]: 2023-06-23T01:09:14.276+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:14.640 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:14 smithi092 conmon[113172]: 2023-06-23T01:09:14.276+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:06.530505+0000 front 2023-06-23T01:08:06.530526+0000 (oldest deadline 2023-06-23T01:09:12.430698+0000) 2023-06-23T01:09:14.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:14 smithi131 conmon[115245]: 2023-06-23T01:09:14.236+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:14.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:14 smithi131 conmon[115245]: 2023-06-23T01:09:14.236+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:15.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:15 smithi092 conmon[113172]: 2023-06-23T01:09:15.237+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:15.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:15 smithi092 conmon[113172]: 2023-06-23T01:09:15.237+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:06.530505+0000 front 2023-06-23T01:08:06.530526+0000 (oldest deadline 2023-06-23T01:09:12.430698+0000) 2023-06-23T01:09:15.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:15 smithi131 conmon[115245]: 2023-06-23T01:09:15.244+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:15.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:15 smithi131 conmon[115245]: 2023-06-23T01:09:15.244+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:16.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:16 smithi092 conmon[113172]: 2023-06-23T01:09:16.251+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:16.640 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:16 smithi092 conmon[113172]: 2023-06-23T01:09:16.251+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:06.530505+0000 front 2023-06-23T01:08:06.530526+0000 (oldest deadline 2023-06-23T01:09:12.430698+0000) 2023-06-23T01:09:16.731 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:16 smithi131 conmon[115245]: 2023-06-23T01:09:16.283+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:16.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:16 smithi131 conmon[115245]: 2023-06-23T01:09:16.283+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:17.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:17 smithi092 conmon[113172]: 2023-06-23T01:09:17.260+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.329652+0000 front 2023-06-23T01:08:01.329817+0000 (oldest deadline 2023-06-23T01:09:05.429668+0000) 2023-06-23T01:09:17.639 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:17 smithi092 conmon[113172]: 2023-06-23T01:09:17.260+0000 7fea7056a700 -1 osd.10 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:06.530505+0000 front 2023-06-23T01:08:06.530526+0000 (oldest deadline 2023-06-23T01:09:12.430698+0000) 2023-06-23T01:09:17.731 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:17 smithi131 conmon[115245]: 2023-06-23T01:09:17.315+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:17.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:17 smithi131 conmon[115245]: 2023-06-23T01:09:17.315+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:18.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:18 smithi131 conmon[115245]: 2023-06-23T01:09:18.340+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:18.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:18 smithi131 conmon[115245]: 2023-06-23T01:09:18.340+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:18.890 INFO:journalctl@ceph.osd.10.smithi092.stdout:Jun 23 01:09:18 smithi092 bash[122158]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-10 2023-06-23T01:09:19.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:19 smithi131 conmon[115245]: 2023-06-23T01:09:19.350+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:19.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:19 smithi131 conmon[115245]: 2023-06-23T01:09:19.350+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:20.105 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.10.service' 2023-06-23T01:09:20.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:09:20.145 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-06-23T01:09:20.145 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-06-23T01:09:20.145 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.11 2023-06-23T01:09:20.482 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:20 smithi131 systemd[1]: Stopping Ceph osd.11 for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:09:20.482 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:20 smithi131 conmon[115245]: 2023-06-23T01:09:20.338+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:20.482 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:20 smithi131 conmon[115245]: 2023-06-23T01:09:20.338+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:20.982 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:20 smithi131 conmon[115245]: 2023-06-23T01:09:20.502+0000 7f07d59cf700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:09:20.982 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:20 smithi131 conmon[115245]: 2023-06-23T01:09:20.502+0000 7f07d59cf700 -1 osd.11 81 *** Got signal Terminated *** 2023-06-23T01:09:20.983 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:20 smithi131 conmon[115245]: 2023-06-23T01:09:20.502+0000 7f07d59cf700 -1 osd.11 81 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-06-23T01:09:21.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:21 smithi131 conmon[115245]: 2023-06-23T01:09:21.335+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:21.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:21 smithi131 conmon[115245]: 2023-06-23T01:09:21.335+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:21.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:21 smithi131 conmon[115245]: 2023-06-23T01:09:21.335+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2023-06-23T01:08:16.562393+0000 front 2023-06-23T01:08:16.562418+0000 (oldest deadline 2023-06-23T01:09:20.662595+0000) 2023-06-23T01:09:22.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:22 smithi131 conmon[115245]: 2023-06-23T01:09:22.317+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:22.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:22 smithi131 conmon[115245]: 2023-06-23T01:09:22.317+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:22.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:22 smithi131 conmon[115245]: 2023-06-23T01:09:22.317+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2023-06-23T01:08:16.562393+0000 front 2023-06-23T01:08:16.562418+0000 (oldest deadline 2023-06-23T01:09:20.662595+0000) 2023-06-23T01:09:23.731 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:23 smithi131 conmon[115245]: 2023-06-23T01:09:23.322+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:23.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:23 smithi131 conmon[115245]: 2023-06-23T01:09:23.322+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:23.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:23 smithi131 conmon[115245]: 2023-06-23T01:09:23.322+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2023-06-23T01:08:16.562393+0000 front 2023-06-23T01:08:16.562418+0000 (oldest deadline 2023-06-23T01:09:20.662595+0000) 2023-06-23T01:09:24.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:24 smithi131 conmon[115245]: 2023-06-23T01:09:24.332+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:24.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:24 smithi131 conmon[115245]: 2023-06-23T01:09:24.332+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:24.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:24 smithi131 conmon[115245]: 2023-06-23T01:09:24.332+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2023-06-23T01:08:16.562393+0000 front 2023-06-23T01:08:16.562418+0000 (oldest deadline 2023-06-23T01:09:20.662595+0000) 2023-06-23T01:09:25.731 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:25 smithi131 conmon[115245]: 2023-06-23T01:09:25.346+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2023-06-23T01:08:01.360059+0000 front 2023-06-23T01:08:01.360219+0000 (oldest deadline 2023-06-23T01:09:07.260112+0000) 2023-06-23T01:09:25.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:25 smithi131 conmon[115245]: 2023-06-23T01:09:25.346+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.92:6804 osd.1 since back 2023-06-23T01:08:09.561484+0000 front 2023-06-23T01:08:09.561353+0000 (oldest deadline 2023-06-23T01:09:13.061410+0000) 2023-06-23T01:09:25.732 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:25 smithi131 conmon[115245]: 2023-06-23T01:09:25.346+0000 7f07cefc7700 -1 osd.11 81 heartbeat_check: no reply from 172.21.15.131:6804 osd.2 since back 2023-06-23T01:08:16.562393+0000 front 2023-06-23T01:08:16.562418+0000 (oldest deadline 2023-06-23T01:09:20.662595+0000) 2023-06-23T01:09:26.482 INFO:journalctl@ceph.osd.11.smithi131.stdout:Jun 23 01:09:26 smithi131 bash[124592]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-osd-11 2023-06-23T01:09:27.709 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@osd.11.service' 2023-06-23T01:09:27.747 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:09:27.748 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-06-23T01:09:27.748 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-06-23T01:09:27.749 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.a 2023-06-23T01:09:28.068 INFO:journalctl@ceph.mds.a.smithi046.stdout:Jun 23 01:09:27 smithi046 systemd[1]: Stopping Ceph mds.a for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:09:28.568 INFO:journalctl@ceph.mds.a.smithi046.stdout:Jun 23 01:09:28 smithi046 conmon[130576]: 2023-06-23T01:09:28.121+0000 7fb4b644f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:09:28.568 INFO:journalctl@ceph.mds.a.smithi046.stdout:Jun 23 01:09:28 smithi046 conmon[130576]: 2023-06-23T01:09:28.121+0000 7fb4b644f700 -1 mds.a *** got signal Terminated *** 2023-06-23T01:09:31.705 INFO:journalctl@ceph.mds.a.smithi046.stdout:Jun 23 01:09:31 smithi046 bash[135521]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-a 2023-06-23T01:09:31.712 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.a.service' 2023-06-23T01:09:31.748 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:09:31.749 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-06-23T01:09:31.749 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-06-23T01:09:31.749 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.d 2023-06-23T01:09:32.068 INFO:journalctl@ceph.mds.d.smithi046.stdout:Jun 23 01:09:31 smithi046 systemd[1]: Stopping Ceph mds.d for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:09:32.568 INFO:journalctl@ceph.mds.d.smithi046.stdout:Jun 23 01:09:32 smithi046 conmon[129166]: 2023-06-23T01:09:32.120+0000 7fe0e4eee700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:09:32.568 INFO:journalctl@ceph.mds.d.smithi046.stdout:Jun 23 01:09:32 smithi046 conmon[129166]: 2023-06-23T01:09:32.120+0000 7fe0e4eee700 -1 mds.d *** got signal Terminated *** 2023-06-23T01:09:35.998 INFO:journalctl@ceph.mds.d.smithi046.stdout:Jun 23 01:09:35 smithi046 bash[135799]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-d 2023-06-23T01:09:36.192 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.d.service' 2023-06-23T01:09:36.226 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:09:36.227 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-06-23T01:09:36.228 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-06-23T01:09:36.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.g 2023-06-23T01:09:36.568 INFO:journalctl@ceph.mds.g.smithi046.stdout:Jun 23 01:09:36 smithi046 systemd[1]: Stopping Ceph mds.g for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:09:37.068 INFO:journalctl@ceph.mds.g.smithi046.stdout:Jun 23 01:09:36 smithi046 conmon[129976]: 2023-06-23T01:09:36.619+0000 7f8dad40f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:09:37.068 INFO:journalctl@ceph.mds.g.smithi046.stdout:Jun 23 01:09:36 smithi046 conmon[129976]: 2023-06-23T01:09:36.619+0000 7f8dad40f700 -1 mds.g *** got signal Terminated *** 2023-06-23T01:09:41.004 INFO:journalctl@ceph.mds.g.smithi046.stdout:Jun 23 01:09:40 smithi046 bash[136082]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-g 2023-06-23T01:09:41.010 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.g.service' 2023-06-23T01:09:41.042 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:09:41.043 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-06-23T01:09:41.044 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-06-23T01:09:41.044 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.j 2023-06-23T01:09:41.318 INFO:journalctl@ceph.mds.j.smithi046.stdout:Jun 23 01:09:41 smithi046 systemd[1]: Stopping Ceph mds.j for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:09:41.818 INFO:journalctl@ceph.mds.j.smithi046.stdout:Jun 23 01:09:41 smithi046 conmon[128594]: 2023-06-23T01:09:41.426+0000 7fe056741700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:09:41.818 INFO:journalctl@ceph.mds.j.smithi046.stdout:Jun 23 01:09:41 smithi046 conmon[128594]: 2023-06-23T01:09:41.426+0000 7fe056741700 -1 mds.j *** got signal Terminated *** 2023-06-23T01:09:45.818 INFO:journalctl@ceph.mds.j.smithi046.stdout:Jun 23 01:09:45 smithi046 bash[136359]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-j 2023-06-23T01:09:45.835 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.j.service' 2023-06-23T01:09:45.868 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:09:45.868 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-06-23T01:09:45.869 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-06-23T01:09:45.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.b 2023-06-23T01:09:46.215 INFO:journalctl@ceph.mds.b.smithi092.stdout:Jun 23 01:09:45 smithi092 systemd[1]: Stopping Ceph mds.b for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:09:46.639 INFO:journalctl@ceph.mds.b.smithi092.stdout:Jun 23 01:09:46 smithi092 conmon[118258]: 2023-06-23T01:09:46.212+0000 7f09f67bd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:09:46.640 INFO:journalctl@ceph.mds.b.smithi092.stdout:Jun 23 01:09:46 smithi092 conmon[118258]: 2023-06-23T01:09:46.212+0000 7f09f67bd700 -1 mds.b *** got signal Terminated *** 2023-06-23T01:09:56.639 INFO:journalctl@ceph.mds.b.smithi092.stdout:Jun 23 01:09:56 smithi092 bash[122706]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-b 2023-06-23T01:09:56.640 INFO:journalctl@ceph.mds.b.smithi092.stdout:Jun 23 01:09:56 smithi092 systemd[1]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.b.service: Main process exited, code=exited, status=137/n/a 2023-06-23T01:09:56.727 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.b.service' 2023-06-23T01:09:56.761 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:09:56.761 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-06-23T01:09:56.762 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-06-23T01:09:56.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.e 2023-06-23T01:09:57.116 INFO:journalctl@ceph.mds.e.smithi092.stdout:Jun 23 01:09:56 smithi092 systemd[1]: Stopping Ceph mds.e for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:09:57.389 INFO:journalctl@ceph.mds.e.smithi092.stdout:Jun 23 01:09:57 smithi092 conmon[117140]: 2023-06-23T01:09:57.112+0000 7f4b7a64a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:09:57.390 INFO:journalctl@ceph.mds.e.smithi092.stdout:Jun 23 01:09:57 smithi092 conmon[117140]: 2023-06-23T01:09:57.112+0000 7f4b7a64a700 -1 mds.e *** got signal Terminated *** 2023-06-23T01:10:01.389 INFO:journalctl@ceph.mds.e.smithi092.stdout:Jun 23 01:10:01 smithi092 bash[122989]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-e 2023-06-23T01:10:01.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.e.service' 2023-06-23T01:10:01.565 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:10:01.566 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-06-23T01:10:01.566 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-06-23T01:10:01.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.h 2023-06-23T01:10:01.889 INFO:journalctl@ceph.mds.h.smithi092.stdout:Jun 23 01:10:01 smithi092 systemd[1]: Stopping Ceph mds.h for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:10:02.389 INFO:journalctl@ceph.mds.h.smithi092.stdout:Jun 23 01:10:01 smithi092 conmon[117697]: 2023-06-23T01:10:01.933+0000 7fe608ce9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:10:02.389 INFO:journalctl@ceph.mds.h.smithi092.stdout:Jun 23 01:10:01 smithi092 conmon[117697]: 2023-06-23T01:10:01.933+0000 7fe608ce9700 -1 mds.h *** got signal Terminated *** 2023-06-23T01:10:04.829 INFO:journalctl@ceph.mds.h.smithi092.stdout:Jun 23 01:10:04 smithi092 bash[123265]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-h 2023-06-23T01:10:04.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.h.service' 2023-06-23T01:10:04.868 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:10:04.869 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-06-23T01:10:04.869 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-06-23T01:10:04.870 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.k 2023-06-23T01:10:05.139 INFO:journalctl@ceph.mds.k.smithi092.stdout:Jun 23 01:10:04 smithi092 systemd[1]: Stopping Ceph mds.k for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:10:05.639 INFO:journalctl@ceph.mds.k.smithi092.stdout:Jun 23 01:10:05 smithi092 conmon[116582]: 2023-06-23T01:10:05.230+0000 7f2b0f4ca700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:10:05.640 INFO:journalctl@ceph.mds.k.smithi092.stdout:Jun 23 01:10:05 smithi092 conmon[116582]: 2023-06-23T01:10:05.230+0000 7f2b0f4ca700 -1 mds.k *** got signal Terminated *** 2023-06-23T01:10:07.639 INFO:journalctl@ceph.mds.k.smithi092.stdout:Jun 23 01:10:07 smithi092 bash[123539]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-k 2023-06-23T01:10:07.651 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.k.service' 2023-06-23T01:10:07.683 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:10:07.684 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-06-23T01:10:07.684 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-06-23T01:10:07.685 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.c 2023-06-23T01:10:07.981 INFO:journalctl@ceph.mds.c.smithi131.stdout:Jun 23 01:10:07 smithi131 systemd[1]: Stopping Ceph mds.c for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:10:08.481 INFO:journalctl@ceph.mds.c.smithi131.stdout:Jun 23 01:10:08 smithi131 conmon[120125]: 2023-06-23T01:10:08.053+0000 7fb99bbdb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:10:08.482 INFO:journalctl@ceph.mds.c.smithi131.stdout:Jun 23 01:10:08 smithi131 conmon[120125]: 2023-06-23T01:10:08.053+0000 7fb99bbdb700 -1 mds.c *** got signal Terminated *** 2023-06-23T01:10:10.731 INFO:journalctl@ceph.mds.c.smithi131.stdout:Jun 23 01:10:10 smithi131 bash[125133]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-c 2023-06-23T01:10:10.850 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.c.service' 2023-06-23T01:10:10.886 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:10:10.887 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-06-23T01:10:10.887 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-06-23T01:10:10.888 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.f 2023-06-23T01:10:11.234 INFO:journalctl@ceph.mds.f.smithi131.stdout:Jun 23 01:10:10 smithi131 systemd[1]: Stopping Ceph mds.f for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:10:11.731 INFO:journalctl@ceph.mds.f.smithi131.stdout:Jun 23 01:10:11 smithi131 conmon[119563]: 2023-06-23T01:10:11.254+0000 7f48c95d6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:10:11.732 INFO:journalctl@ceph.mds.f.smithi131.stdout:Jun 23 01:10:11 smithi131 conmon[119563]: 2023-06-23T01:10:11.254+0000 7f48c95d6700 -1 mds.f *** got signal Terminated *** 2023-06-23T01:10:15.732 INFO:journalctl@ceph.mds.f.smithi131.stdout:Jun 23 01:10:15 smithi131 bash[125414]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-f 2023-06-23T01:10:15.863 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.f.service' 2023-06-23T01:10:15.895 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:10:15.896 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-06-23T01:10:15.896 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-06-23T01:10:15.896 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.i 2023-06-23T01:10:16.231 INFO:journalctl@ceph.mds.i.smithi131.stdout:Jun 23 01:10:15 smithi131 systemd[1]: Stopping Ceph mds.i for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:10:16.731 INFO:journalctl@ceph.mds.i.smithi131.stdout:Jun 23 01:10:16 smithi131 conmon[120686]: 2023-06-23T01:10:16.277+0000 7f71e1dc5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:10:16.732 INFO:journalctl@ceph.mds.i.smithi131.stdout:Jun 23 01:10:16 smithi131 conmon[120686]: 2023-06-23T01:10:16.277+0000 7f71e1dc5700 -1 mds.i *** got signal Terminated *** 2023-06-23T01:10:19.161 INFO:journalctl@ceph.mds.i.smithi131.stdout:Jun 23 01:10:18 smithi131 bash[125694]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-i 2023-06-23T01:10:19.168 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.i.service' 2023-06-23T01:10:19.198 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:10:19.199 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-06-23T01:10:19.199 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-06-23T01:10:19.200 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.l 2023-06-23T01:10:19.482 INFO:journalctl@ceph.mds.l.smithi131.stdout:Jun 23 01:10:19 smithi131 systemd[1]: Stopping Ceph mds.l for 458642ee-1161-11ee-9b2b-001a4aab830c... 2023-06-23T01:10:19.981 INFO:journalctl@ceph.mds.l.smithi131.stdout:Jun 23 01:10:19 smithi131 conmon[119005]: 2023-06-23T01:10:19.575+0000 7f59911e9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-06-23T01:10:19.982 INFO:journalctl@ceph.mds.l.smithi131.stdout:Jun 23 01:10:19 smithi131 conmon[119005]: 2023-06-23T01:10:19.575+0000 7f59911e9700 -1 mds.l *** got signal Terminated *** 2023-06-23T01:10:21.231 INFO:journalctl@ceph.mds.l.smithi131.stdout:Jun 23 01:10:21 smithi131 bash[125971]: ceph-458642ee-1161-11ee-9b2b-001a4aab830c-mds-l 2023-06-23T01:10:21.278 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-458642ee-1161-11ee-9b2b-001a4aab830c@mds.l.service' 2023-06-23T01:10:21.310 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-23T01:10:21.311 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-06-23T01:10:21.311 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 458642ee-1161-11ee-9b2b-001a4aab830c --force --keep-logs 2023-06-23T01:10:23.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 458642ee-1161-11ee-9b2b-001a4aab830c --force --keep-logs 2023-06-23T01:10:25.397 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 458642ee-1161-11ee-9b2b-001a4aab830c --force --keep-logs 2023-06-23T01:10:27.376 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-23T01:10:27.409 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-23T01:10:27.438 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-06-23T01:10:27.465 INFO:tasks.cephadm:Archiving crash dumps... 2023-06-23T01:10:27.468 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908/remote/smithi046/crash 2023-06-23T01:10:27.469 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/crash -- . 2023-06-23T01:10:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:tar: /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/crash: Cannot open: No such file or directory 2023-06-23T01:10:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:tar: Error is not recoverable: exiting now 2023-06-23T01:10:27.521 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908/remote/smithi092/crash 2023-06-23T01:10:27.522 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/crash -- . 2023-06-23T01:10:27.578 INFO:teuthology.orchestra.run.smithi092.stderr:tar: /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/crash: Cannot open: No such file or directory 2023-06-23T01:10:27.578 INFO:teuthology.orchestra.run.smithi092.stderr:tar: Error is not recoverable: exiting now 2023-06-23T01:10:27.579 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908/remote/smithi131/crash 2023-06-23T01:10:27.581 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/crash -- . 2023-06-23T01:10:27.614 INFO:teuthology.orchestra.run.smithi131.stderr:tar: /var/lib/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/crash: Cannot open: No such file or directory 2023-06-23T01:10:27.614 INFO:teuthology.orchestra.run.smithi131.stderr:tar: Error is not recoverable: exiting now 2023-06-23T01:10:27.615 INFO:tasks.cephadm:Checking cluster log for badness... 2023-06-23T01:10:27.616 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-06-23T01:10:27.645 INFO:teuthology.orchestra.run.smithi046.stderr:grep: /var/log/ceph/458642ee-1161-11ee-9b2b-001a4aab830c/ceph.log: No such file or directory 2023-06-23T01:10:27.647 INFO:tasks.cephadm:Compressing logs... 2023-06-23T01:10:27.648 DEBUG:teuthology.orchestra.run.smithi046:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:10:27.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:10:27.692 DEBUG:teuthology.orchestra.run.smithi131:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:10:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-06-23T01:10:27.714 INFO:teuthology.orchestra.run.smithi092.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-06-23T01:10:27.716 INFO:teuthology.orchestra.run.smithi131.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-06-23T01:10:28.774 INFO:tasks.cephadm:Archiving logs... 2023-06-23T01:10:28.774 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908/remote/smithi046/log 2023-06-23T01:10:28.776 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T01:10:28.967 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908/remote/smithi092/log 2023-06-23T01:10:28.969 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T01:10:29.087 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908/remote/smithi131/log 2023-06-23T01:10:29.096 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T01:10:29.215 INFO:tasks.cephadm:Removing cluster... 2023-06-23T01:10:29.216 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 458642ee-1161-11ee-9b2b-001a4aab830c --force 2023-06-23T01:10:30.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 458642ee-1161-11ee-9b2b-001a4aab830c --force 2023-06-23T01:10:30.796 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 458642ee-1161-11ee-9b2b-001a4aab830c --force 2023-06-23T01:10:31.581 INFO:tasks.cephadm:Removing cephadm ... 2023-06-23T01:10:31.582 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/cephadm 2023-06-23T01:10:31.599 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf /home/ubuntu/cephtest/cephadm 2023-06-23T01:10:31.615 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf /home/ubuntu/cephtest/cephadm 2023-06-23T01:10:31.630 INFO:tasks.cephadm:Teardown complete 2023-06-23T01:10:31.631 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 1561, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 1474, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 896, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true' 2023-06-23T01:10:31.716 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5de3ac8d371442f3a5984c6b23f1d4af Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 1561, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 1474, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephadm.py", line 896, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true' 2023-06-23T01:10:31.723 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-06-23T01:10:31.738 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-06-23T01:10:31.746 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-06-23T01:10:31.747 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T01:10:31.760 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T01:10:31.763 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T01:10:31.870 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-06-23T01:10:31.879 INFO:teuthology.task.clock:Checking final clock skew... 2023-06-23T01:10:31.879 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:10:31.882 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:10:31.884 DEBUG:teuthology.orchestra.run.smithi131:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:10:31.895 INFO:teuthology.orchestra.run.smithi046.stderr:bash: ntpq: command not found 2023-06-23T01:10:31.897 INFO:teuthology.orchestra.run.smithi092.stderr:bash: ntpq: command not found 2023-06-23T01:10:31.899 INFO:teuthology.orchestra.run.smithi046.stdout:210 Number of sources = 4 2023-06-23T01:10:31.899 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:10:31.900 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2023-06-23T01:10:31.900 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +2122us[+2123us] +/- 87ms 2023-06-23T01:10:31.900 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -1873us[-1872us] +/- 80ms 2023-06-23T01:10:31.901 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 70 +33us[ +33us] +/- 85ms 2023-06-23T01:10:31.901 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:10:31.902 INFO:teuthology.orchestra.run.smithi131.stderr:bash: ntpq: command not found 2023-06-23T01:10:31.903 INFO:teuthology.orchestra.run.smithi092.stdout:210 Number of sources = 4 2023-06-23T01:10:31.903 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:10:31.903 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2023-06-23T01:10:31.903 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 +2165us[+2165us] +/- 87ms 2023-06-23T01:10:31.904 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -1833us[-1837us] +/- 80ms 2023-06-23T01:10:31.904 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +85us[ +82us] +/- 86ms 2023-06-23T01:10:31.904 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:10:31.905 INFO:teuthology.orchestra.run.smithi131.stdout:210 Number of sources = 4 2023-06-23T01:10:31.906 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:10:31.906 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2023-06-23T01:10:31.906 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 +2074us[+2074us] +/- 87ms 2023-06-23T01:10:31.907 INFO:teuthology.orchestra.run.smithi131.stdout:^* hv02.front.sepia.ceph.com 3 6 377 7 -1904us[-1908us] +/- 80ms 2023-06-23T01:10:31.907 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 -2204ns[-2204ns] +/- 86ms 2023-06-23T01:10:31.907 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:10:31.908 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-06-23T01:10:31.939 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-06-23T01:10:31.940 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-06-23T01:10:31.950 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T01:10:32.050 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T01:10:32.139 DEBUG:teuthology.orchestra.run.smithi131:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T01:10:32.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:10:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:10:32.261 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:10:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:10:32.291 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:10:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:10:32.321 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-06-23T01:10:32.332 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-06-23T01:10:32.343 INFO:teuthology.task.internal:Duration was 1539.006066 seconds 2023-06-23T01:10:32.344 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-06-23T01:10:32.353 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-06-23T01:10:32.354 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T01:10:32.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T01:10:32.359 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T01:10:32.401 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:10:32.402 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:10:32.405 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:10:32.651 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-06-23T01:10:32.652 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2023-06-23T01:10:32.652 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T01:10:32.673 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:10:32.674 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T01:10:32.712 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2023-06-23T01:10:32.712 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T01:10:32.732 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-06-23T01:10:32.732 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:10:32.735 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:10:32.755 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:10:32.816 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-06-23T01:10:32.817 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T01:10:32.820 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T01:10:32.836 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T01:10:32.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-06-23T01:10:32.951 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-06-23T01:10:32.951 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T01:10:32.979 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T01:10:33.004 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T01:10:33.029 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-06-23T01:10:33.039 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:10:33.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:10:33.046 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:10:33.066 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2023-06-23T01:10:33.067 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2023-06-23T01:10:33.092 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2023-06-23T01:10:33.106 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:10:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:10:33.136 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:10:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:10:33.150 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:10:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:10:33.164 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-06-23T01:10:33.174 INFO:teuthology.task.internal:Transferring archived files... 2023-06-23T01:10:33.175 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908/remote/smithi046 2023-06-23T01:10:33.175 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T01:10:33.212 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908/remote/smithi092 2023-06-23T01:10:33.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T01:10:33.245 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908/remote/smithi131 2023-06-23T01:10:33.246 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T01:10:33.286 INFO:teuthology.task.internal:Removing archive directory... 2023-06-23T01:10:33.286 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T01:10:33.289 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T01:10:33.291 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T01:10:33.330 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-06-23T01:10:33.348 INFO:teuthology.task.internal:Not uploading archives. 2023-06-23T01:10:33.349 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-06-23T01:10:33.361 INFO:teuthology.task.internal:Tidying up after the test... 2023-06-23T01:10:33.361 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T01:10:33.363 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T01:10:33.365 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T01:10:33.376 INFO:teuthology.orchestra.run.smithi046.stdout: 262176 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 01:10 /home/ubuntu/cephtest 2023-06-23T01:10:33.377 INFO:teuthology.orchestra.run.smithi092.stdout: 262149 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 01:10 /home/ubuntu/cephtest 2023-06-23T01:10:33.385 INFO:teuthology.orchestra.run.smithi131.stdout: 262150 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 01:10 /home/ubuntu/cephtest 2023-06-23T01:10:33.386 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-06-23T01:10:33.450 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-06-23T01:10:33.504 INFO:teuthology.nuke:Checking targets against current locks 2023-06-23T01:10:33.531 DEBUG:teuthology.nuke:shortname: smithi046 2023-06-23T01:10:33.532 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:10:33.557 DEBUG:teuthology.nuke:shortname: smithi092 2023-06-23T01:10:33.557 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:10:33.561 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:37:21.158050', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN1AP+f9En/qWCTR517bW3K2TEIx74WvyU0+UStEBwjA4HOA+Dk6quiGXlPiyJiQLebEZpT7q6txD3U7TWt2yw7R0ASa4rIaQQ5eD17stmLN8AbYAqo3HnMFe+/arNslnB2u3mO3A4Cv5GvnQDxSnRCyJPgprPV8Ohsle9rfQcnF2iFigaAJluuHxMhEgfq4dSQ+cWZO0y/2AF3wfchXJXnPZf6ePR+mNyqYkrBnVoZy3pXlDhI5mRQN0WxcEFWcekaYi53sf6v1RDamz50QMKoOqdWD955PL8h66luT2Gm7nM/qOqtvHCC8tbFxlqsIpQgJ/39LjZNQXplFpms1AaJqRMk7gUAY9RSG9JdkW7oqglr0N7GzemCKoNuOGYptFRWsUkM5T+mqujhRPIwgZGFAoreeRHmf8vre8oYz/gy+qsYe2hk5eHuIIB0j/xdYu5ayfkOCqXFa6pyejJNI/SE7kpRDexHtONnhmnDK9p6PRD+rEEAnB8vJ4e5QlfoEk='} 2023-06-23T01:10:33.578 DEBUG:teuthology.nuke:shortname: smithi131 2023-06-23T01:10:33.578 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:10:33.588 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:37:21.156577', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJrLrBzXWVmQ3L1uZgHMn/snozkRm3Cwl6ZcJjgQX3uwpqIHAHSgbISAqsC2Hq8cVRoNT/YeDmx0bg9WR8bTItw='} 2023-06-23T01:10:33.598 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311908', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:37:21.159316', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnDHs2ZAbaoFqu4S0M5u5Efu8EYNCAAOCkrnxRvUtNDTXZ9v7+gc4/D2UifAgtA2MWSXlTfVfWLbQ8giOkgulaOgsGSQ1PJUul1xefDvNiGsbr0gajnQ/SMB68P4R31eukuYf6BiMIZs6onNLqJzotTtMDV5hhSs6LakIbxzUMeyiW95TImavGgv2hF6esNLyefFLi3TQTl8gUma6LBeaOzTu8VMevCnrzyFd7XcTvd42vlY77zzPtTZbCDPocI59botf9ajpTe8Mkbwn+JHxCkGAiZWuzxM0ZNZ0D2OxR7XYbqHYOQxQLYlxMXIr13y4X087xo9rGbzPGXAXtUVfwZnQd3KqYqu5VgIcvqX6nlGvHGvtVwhs4ATbtzV9VHubyiJC6u0ZjYy7amMLIrLNFYIgDzHXRkGBm0xu9ftVvrJfJtkz/VQiEOEujTZBuCL5/6SqRApvNI+9K/W3iA5tzC/y1plO2MEZzmpW3b3B2mDVaLmbfCpbObPltoDhljSc='} 2023-06-23T01:10:33.612 INFO:teuthology.orchestra.console:Power off smithi046 2023-06-23T01:10:33.613 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T01:10:33.639 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:10:33.660 INFO:teuthology.orchestra.console:Power off smithi092 2023-06-23T01:10:33.661 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T01:10:33.678 INFO:teuthology.orchestra.console:Power off smithi131 2023-06-23T01:10:33.679 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T01:10:33.690 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:10:33.706 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:10:37.691 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:10:37.717 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:10:37.732 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:10:41.808 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:10:41.830 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:10:41.846 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:10:42.033 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2023-06-23T01:10:42.042 INFO:teuthology.orchestra.console:Power off for smithi092 completed 2023-06-23T01:10:45.960 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:10:46.172 INFO:teuthology.orchestra.console:Power off for smithi131 completed 2023-06-23T01:10:46.277 INFO:teuthology.run:Summary data: description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} duration: 1539.0060663223267 failure_reason: 'Command failed on smithi046 with status 22: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5de3ac8d371442f3a5984c6b23f1d4af status: fail success: false 2023-06-23T01:10:46.278 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T01:10:46.362 INFO:teuthology.run:FAIL